Niall Crosby, founder of AG Grid, appeared on the JavaScript Jabber podcast Episode 504, released on October 12th 2021. Topics discussed included - JavaScript Data Grids, supporting multiple frameworks, DOM performance, Web Components, IOC, developing with no dependencies, turning open source into a profitable business model, typescript and more.
Listen to the Full JavaScript Jabber Episode
This is an edited transcript of Niall Crosby's interview on JavaScript Jabber Episode 504 "AgGrid: From Open Source to Successful Business". Check out the podcast if you want to hear the full conversation with contributons and comments from all the hosts, including the many jokes and learning resource references.
You can listen to the full episode on the JavaScript Jabber Podcast site, or from your favourite podcast app.
- JavaScript Jabber Episode 504
- Episode Hosted by:
- AJ O'Neal
- Charles Wood
- Dan Shappir
- Steve Edwards
What is AG Grid
You can split JavaScript developers into kind of two different categories. You've got the people who are building enterprise applications, which is building apps to help run businesses. And then you've got people who are building consumer facing web portals, and they're a much bigger group.
On the consumer facing web portals, you normally don't use data grids. If you want to display data on a web page in tabular format, you'd normally just use HTML table, that's all you need. But if you're building apps for enterprise, then it's very common to have a data grid in your application, and doing functions.
If you imagine that a trader inside a bank is looking at loads of stock prices, he wants a very rich experience with his data, and he ideally wants to be able to do the types of things you do in Excel: you want to sort columns, resize them, move them around, do filtering, then do pivoting aggregations, maybe go to a chart, all of that type of stuff.
AG Grid is, is a library that allows developers to plug in a rich grid experience into their application.
So we're a third party library widget.
How was AG Grid Created?
I got into the space because I was working for the banks in London for about maybe five or six years. And there was one particular project back in 2014, where the requirements were such that I couldn't build what I wanted to build with the data grids that were available at that time.
JavaScript and enterprise was kind of new and the libraries available were still maturing so out of frustration I built ag grid as an open source project.
One Christmas as a challenge to myself two weeks. So if I pulled up the laptop and went, hey, how hard could this be?
And here I am seven years later. But what was interesting though, is even within two weeks, I had something which was differentiated itself enough from what was already available both in the community and in the paid for versions as well.
What were other grids missing?
Well, I think if I look back what was available at the time you couldn't pin columns, sometimes called frozen columns, where you just want a column to stick to the left hand side as you're as you're scrolling, right and left. And that's a very basic feature.
Also they couldn't do row grouping, if you wanted to drag a column and say, group by by this row. From a feature point of view, a lot of the grids were just feature incomplete.
And what I believe the reason why that is, is because people who come up and they create a data grid, and it solves one problem very well, everybody would get the row virtualization working, then maybe put in row grouping, but then things would begin to lose quality as the added complexity grows.
I think that's the number one feature that all these data grids need to have. That's where you jump from a table into what I would consider an enterprise data grid. You have to solve row virtualization. And then we do call them virtualization as well.
So you've got maximum benefits of keeping the DOM light.
Building a grid is hard because as they add features, the complexity goes up exponentially. The reason for that is in a data grid, when you add a feature, it has to work with all the features that came before.
When I implemented the first grid that did moving, implemented sorting that worked, then I implemented filtering that needed to work with sorting as well. And then I needed to implement pins columns, which needed to work with sorting, and filtering, then we went to row grouping that had to work with all the features before.
That complexity of introducing features working together does go up exponentially. And that's why we have seen a lot of grids that start off well which solve specific problems well, but then they struggle to get the complete feature set to be able to provide for a general purpose data grid.
What is Row Virtualization?
If you, for example, have 10,000 rows, with about 20 columns? You can forget tables, if you try and render 10,000 rows into the DOM with 20 columns, you use an HTML table, it just hangs.
What you can do is load the data from your server and have 10,000 rows in memory, and put them into a scrollable, viewport. And then as the user is scrolling up and down, you just draw the DOM for what the user can actually see. So if you're using divs, to create rows, then you've only got 10 divs existing if you can only see 10 in a viewport. And then as user scrolls down, you create more divs and remove the ones that were there previously.
We maintain proper sizing for the scroll bar because the container that these rows are in is the correct height.
If you have 10,000 rows, and you're 10 pixels high, then your DIV is 100,000 pixels high. And that gives you the full span of the scroll.
And that's called row virtualization.
Was the grid ever re-architected?
We are constantly prototyping, and constantly rewriting. I think I've rewritten the core of the grid maybe four times now. And there's lots of other areas we keep rewriting. Also, when this started, it was a small project.
When it's just a pet project, you just don't have to pick from any rules. So I chose no framework. And I chose no other external libraries. So right from the start, the grid has zero dependencies, which is done because I wanted to find out what it would be like for my own personal interest.
It transpires, that that was a brilliant idea.
Big organisations love the fact that we've got zero dependencies, because if you want to onboard into a larger organisation, if you have dependencies, they not only have to onboard you to have to onboard all of your dependencies as well.
It is such a breath of fresh air for them, when they say, Oh, you've got no dependencies that makes onboarding now, much easier.
But having done that, having no libraries, I didn't have an IOC container. I did't have dependency injection, and that caused issues for me.
The first version of Ag grid, because it didn't have dependency injection, managing references to objects was something that was complex. And I kind of went through this journey where I really felt the pain of not having an IOC container, I mean, an inversion of control container.
Because I didn't have one of them, I had to write my own, which then led to a new version of AG Grid, where it's then built on what we call AG Stack.
So internally, in AG Grid, we now have our own IOC container. And we have our own component library and rendering engine that AG Grid is built on, and that evolved over time.
What is Inversion of Control?
If you're using a framework, then you're probably familiar with managing services inside that framework.
Angular, for example, has services, where it'll allow you to provide a class, it'll instantiate an instance of that class. And then you can inject other services into that class. And then likewise, if you have a component, you can inject services from your application into components.
React and Vue, they all have similar mechanism for the purpose.
The services an IOC container provides is, well, it's twofold.
If your application requires an instance, then the framework will instantiate the instance for you. And then it will manage providing references to other instances, in that instance, that just created, if you don't have that, then your application will have to have a boot sequence that you write yourself, whereby you have all these new methods are certified, and you have all these new calls, where you're creating new instances of all these objects that your application needs. And then you'll have the second step, where you start providing references around to all these different objects that need references to each other. And then when you create a components, which wouldn't be a service, then that needs to make it references to all these objects that it needs. So in an application, a typical object would be the user authentication service or the HTTP service or something like that.
Without IOC we have to write boilerplate code. And boilerplate code is bad. History has taught us that if you can get frameworks to remove the boilerplate code, then that's good. It simplifies your code. And it also forces consistency across your codebase.
Antoher reason we have IOC containers is for testing. So IOC containers is what enables a test library to instantiate an instance of the service you're trying to test. And then it can then inject proxies for that dependency so that you can test that service without requiring a full instance of your application.
AG Grid is dependency free
When I first wrote the first version of Ag grid, I didn't have any libraries, and I still don't have any libraries. So over time, we had to create the libraries that one would be used to if you're using something like Angular or React.
We've kept it completely dependency free, and a reason for that is obviously we're supporting different frameworks. So we support Angular, we support React, we support Vue.
And if a Angular Person person is using AG Grid, they don't want React libraries. And likewise, if someone is using React they don't want to see Vue or Angular. So it was very important for us to keep it as clean as possible and have no dependencies.
How did AG Grid move from an Open Source project to a company?
I created this project, and I created for myself. And when I went back to work after Christmas, I wanted to use it in my project. But I didn't want to just give them the project. Because I wrote it myself, I wanted to keep the IP.
The only way I could think of doing that at the time was to release the project as open source. And then make the project look like an open source project for my employer.
I created a website with some documentation. So it looked like a proper bona fide open source project. So then when I came to them and said, Look, I've created this data grid, it's open source, it's better than the other options out there. Can we use this in our project and work? And they said yes.
Over the next while I was playing two roles, or two hats. During the day, I was a developer inside a project for my employer. And then in the evenings and weekends, I was developing AG Grid.
So I was coming up with requirements during the day. And then in the evenings and weekends programming to try and get it into work.
Because I had a website. And because it was in GitHub, other people started to take notice.
And that wasn't part of the plan.
People started sending me messages and GitHub and doing pull requests.
And what I didn't get. It took me a few months to get this.
But my problem wasn't personal. It wasn't just my problem. There was a global need for the data grid.
The issues and frustrations I had with data grids at the time were shared, lots of people had those.
So I was getting messages saying Do you realise what you've just done, you've just created something that there's a big need for.
So on March the 16th in 2015, I wrote a blog called why the world needed another Angular JS grid, because I was working with Angular at the time. I wrote a blog. And it was read.
By the way, I've got no social footprint at this point. I wasn't a popular speaker at conferences, I didn't even have a Twitter account. There's no social footprint.
I created a blog on medium, and the blog went viral. It was read 6000 times in the first 24 hours and from that day onwards, about 500 people came through the website.
That's when this thing really started going places it started basically being a train moving. And from then there's been this huge story to where we are now we're just train it's just gotten faster and faster and faster on bigger and bigger and bigger.
By, say after that first year, where I was working in my job during the day and do an ag grid. And during the evenings and weekends, I was trying my best to have relationship with my girlfriend. And it was just too much something to give. I couldn't continue the grid and do everything else in my life.
I kept the girlfriend, I dropped the job.
Okay, I quit my job. And you know what? I'm going to give this one year. Let's see if I can get to the next year was 2016. I made a promise to myself "can I get to the end of 2016? and not have my bank balance dropped?"
So basically, can I pay my mortgage? Go to restaurants once or twice a week and just have a nice normal life? Could it get to that self sustain itself?
By the end of that first year I released ag grid enterprise, which was the same plus more in terms of features and also paid support. And by the end of the first year I had two employees. So that only was like cash positive.
I was able to hire people and it was the start of a bonafide company.
Just to kind of put it in context of where we are now. Today we've got '000s of company customers and that's not individual developers. If you think about a company in the fortune 500, that's just one customer out of our seven or 8000.
And about 80% of the Fortune 500 are currently customers of ours at some capacity.
And then some companies we've got site deals with. If you Google top 10 investment banks globally, half of them have basically said we're standardising on AG Grid,
What is an open source business model?
We just try and keep it simple. So the community version is, is proper open source, anybody can use it, and never need to give us a penny or even tell us that you're using it.
Then the enterprise version comes with more features. If you're using the enterprise version, you pay us some money to get more features, and access to our support system.
I think that that works really well for us, because the free version that we have, still stands up there, and is very strong competition against not only the free grids that are out there, but even the enterprise grades. If you think of our competitors, what we give away for free, they're effectively charging for.
Where we charge is where we're going above and beyond what they agreed to typically do.
So one of the first things we implemented that was above and beyond was pivoting. There's no other general purpose data grid out there that will allow you to pivot the data.
There are some specific grids to do pivoting. But then they won't let you do all these other things like sorting and grouping.
That's what a lot of our competitors do. They have four or five different data grids like a tree grid, or pivot date grid or Excel grid or grouping grid.
We have all the features just in one grid, and includes pivoting. So it's not a case where we're charging for something that you could get free elsewhere, or you could pay for elsewhere, we are genuinely doing stuff, which is above and beyond what everybody else is doing.
AG Grid has a charting library?
It's built on top of Canvas. To avoid the performance issues with complex SVG. Not for everybody will have complex charts. But you will have a limit to how complex your charts can get.
One thing I love about charts is, so this is fresh in my head, because we're speaking about where we've gone above and beyond what the competition is doing.
I'm very excited about our charts. And we give our standalone charting library away for free. And the reason why we do that is we think we're doing a charting library, that's just as good, possibly better than everybody else's. But it hasn't exploded onto the scene and done things so much better. So that's why we give that away for free.
But where we have exploded onto the screen onto the scene and have done something way, way, way better is the integration between the two.
If you use AG Grid enterprise, you can just select a range of cells, right click and send to chart and you have an Excel type experience inside your data grid.
Without having to write any code, you just set a property on the grid 'enable charts true', and you have this rich charting experience.
If you were to use any other library, you would need to get a grid and a charting library independently, and then write a lot of code to create that application experience.
Nobody else is doing integrated charting. And that's why we think that that's what should go into our enterprise version because we've done something which is just truly world first and groundbreaking.
Watch the Charting Library Overview Video on YouTube
Monetising Open Source projects
It is difficult. There's a common expectation amongst developers that things should be free, which is changing.
And in recent years, we have patron systems in place where people can donate to projects, but it is a struggle for open source developers, if they're going to rely on donations.
Like, let me give you an example of before we started charging for AG Grid. So back in 2015, I had a Donate button on my website. And over that full year, I got about maybe 10 or 12 donations, and in total, I made about 500 pounds British pounds $800 in donations that year.
Which is a nice gesture, but it's not going to pay the rent.
Once we went enterprise, on the first day, like I was charging 250 pounds a licence when we first launched, and I started selling to maybe a team a day, a four person team.
So right from the start, it was getting about 1000 pounds a day in from customers, which was more than my entire amount of donations the year before.
These are the same people, the people who were buying from me at the early stages, where it was that community of people who were using he grid, since its inception, and were champions of the project, they loved it but the Donate button was mostly just ignored.
But as soon as I had this option for them to buy something from me, lots of them come out of the woodwork. And these customers we never knew I had up until then, they were just NPM downloads.
And banks from all over the world started coming forward and saying, yes, we've been using you for X number of months, we're fans of yours, we'd love to buy 10 licences.
So my takeaway from it is, people are generally not good at taking money out of their back pockets and giving it to you, but they're very good at asking for an invoice and going to their boss and getting an invoice signed off for a much larger amount.
If you are an open source project, and you want to monetize, if you can find a way to sell something, as opposed to depending on contributions, then people in the community will be better able to respond and start giving you money.
Enterprise customers want longevity of the company. They want to make sure you're in business in five years. And they want support.
The typical open source lifecycle is it's great for a couple of years, and then it dies off because the initial creators of the open source project move on in life. They have kids or they move jobs or something like that.
One of the first customers I had was one of the first sales that was very, very weird for me, looking back, I am so happy that it happened.
I was charging 250 pounds per developer and this gentleman and he really was a gentleman said, "Okay, I've got five developers, so you're going to charge me 1250 pounds".
And I said "yes".
And he said, "and I can use the data grid to create for a year, and then if I don't want to continue with support, I can, that's all the money I ever need to give you".
And I said, "Yep".
And he said, "that's not enough, you're not charging me enough. What I'm going to do is I'm going to pay you 10,000 euros, and I want you to go and get a proper Eula".
Because the Eula that I had wasn't very good I copied somebody else's and renamed their name.
"And I want you to start charging more, and set up a proper company, because we have a huge dependency on AG Grid in our application, and we're a proper company as well. But we need to make sure that you stay in business for the next 5-10 years. And you're not going to be able to do that with the small amount you're charging."
It was a strange phone call, because I ended up with my salesperson who may have my opening price and he ended up giving me 10 times more. But I learned so much. And it really gave me an appreciation for why people want you to be a proper company.
They want to pay money, because they want you to be around and be a business partner with them for the future.
Handling Pull Requests from users?
The pull requests that we get are very basic. There were more pull requests towards the start. And the there was a few times where people would implement a feature. But what I found with the pull requests is people writing the pull requests, were only catering or looking at for themselves.
So they would implement a feature, and not fully test it against the complete feature set of the grid, but only test it against the features that they were using.
And there was one example of this where somebody wanted to have column groups, I think there's a more common thing, basically, if you want to have two levels of groupings, so you're grouping your columns into more groups. And he implemented a version of this that worked for him. But it just didn't work with lots of other features like moving columns around, it just completely broke down, because that's not a feature that he wanted.
And then also the fact that there's been work, you're talking about coding styles, the direction of the project, AG Grid is a very complex project.
And I'm a huge believer in colocated people working together, all the people who work for AG Grid now are in London, and the dev team sits together in one room. And even when a new joiner starts in AG Grid, it takes about four to six months to get to know the codebase, there's a big steep learning curve, and it takes them a long time to get to that level where they can comfortably contribute.
So the grid's been at the stage now, and has been for a couple of years where people just don't don't provide pull requests because the project is too complex.
What we do get is small bug fixes, and they are brilliant, we do genuinely get some small bug fixes.
But when you talk about implementing new features, we just don't get those pull requests.
The cost of Ag grid is tiny compared to a day rate over the year. So I think at the moment we're between $750 to $1,200 per developer, which is one day, give or take for development in the United States. People don't spend multiple days trying to add features. it would take you months just to learn our code base that mind that features so it just so just doesn't make sense for you to invest time when you can just pay with a minimal amount.
Transitioning from JavaScript to TypeScript?
Definitely TypeScript, definitely typed. I'm a Java developer, which is not true anymore, because I've been doing AG Grid for about seven years.
I used to be a Java developer. And when I was working for these banks we were using JavaScript, and Angular was the front end, which I just barely knew.
I saw AG Grid as a good way for me to learn JavaScript because I wanted to get better.
I started the project as plain JavaScript, because it's just a pet project. Why bother? But after a few months, I really craved for proper typing, because there's so many benefits to TypeScript when you're managing a large project.
As soon as I couldn't remember the different interfaces, and I wanted code completion and compile time checking, I moved to TypeScript about four months into the project lifecycle.
Remaining Hands on coding and growing a company
You know, one thing I miss is jobs.
Because when you're employed you're thinking about the next job and your CV and you're learning new technology and you're thinking about how you would come across in the next job interview trying to explain this technology.
But I think I've my experience over the last seven years I've basically made myself unemployable from my dream job because now like "CEO of this company with 30 people and you know, given technical direction, blah, blah, blah".
But I love to program. That's just what I really like doing. It's my passion, like a lot of developers, but I just can't see myself being accepted as a developer role anymore.
As the company grew well at the start, I was doing everything because it was just me I was doing sales and marketing and doing blogging, I was writing all the code managing everything, and then was all about backfilling.
It's about finding people to take on specific roles.
So get another developer to help me with development, but then hire somebody to just do sales, hire somebody to just do support. And that grew and grew and grew.
I was always involved in all the different areas because I was so passionate, about it and wanted to be involved.
Even though I wasn't doing the sales, I wanted to know what was happening and helping people and guiding them.
But the company just then got to the stage where there was just too much going on, there was just too much, too many people, too many events, too many options, too many decisions.
And that's where I had to just change the dynamics of the company, and start putting people responsible for certain areas.
So right now I've got what I think is the ideal for me. There's three departments in the company. Basically, we've got sales, tech, and support. And each of those three departments as a department head, and they're responsible for any of that department. With those three people, if they have any questions, they'll come to me, but they're running departments on a day to day basis. And one of those departments have said was the technology department.
I've got a guy, brilliant person, Rob Clark. And he's actually head of product development and the technology department. He is in charge of the developments in AG Grid, and I'm in his team. So I'm actually a senior developer, reporting into Rob. And then I'm also the CEO, having Rob and the other guys reporting into me.
That's messy. But what gives us direction or keeps us in the right direction, as we both understand what the end goal is, we're both working towards the same place like that this project.
It's my baby. So it's very difficult to replace my experiences with AG Grid over the last seven years. It's designed exactly as I would design it, because I did design it. So it's difficult to backfill my role on the grid. So it's from, say, a project point of view, either on the big assets to the company, being the senior developer on the grid. But then on top of all that, I have to run the company as well. So we isolate those concerns, and plus I get to be a developer, because I'm happy being a developer.
Where will AG Grid be in 7 years?
I've learned over the last seven years how difficult writing a data grid is. You might think it's difficult, but trust me, it's far more difficult than you think.
Because I felt it was difficult five years ago. And every year I realised, oh my god, this is so much more, so much more, so much more.
We had ideas in the past about how we could pivot the business to come up with a new project and we had to just stop thinking about those things. Because the grid was just taking so much energy, like even now, seven years later, and I'm very proud of what we've done the grid, but there's so much more we can do, like all these feature requests coming from customers.
Plus keeping up to date with the frameworks like React hooks came out last year or the year before. And we had to rewrite a very large section of the grid to make it work better with react hooks. So just maintaining the grid and the charts and moving them forward is a massive project and we're not out of the woods yet with them.
I would like to think that in seven years from now, we haven't taken our eye off the ball that we've continued to extend the grid and the charts and make them even better than they are today.
If I was to think has the company going to change what new stuff would come out, I'm afraid to say these things because they may not come true. But we would like to develop out the story of other reporting replacements.
If you think enterprise and enterprises want to build applications, it's common for them to also use reporting tool and provide reports to fit alongside the application.
It's also common for somebody using an application to export to Excel, and do some fidgeting with the data to prepare a chart or something to put into an email or a presentation.
What we would like to do is stop people leaving your application, if somebody exports to excel and moves to a BI tool, then your application has failed. Because you haven't met the user requirements of the user, they're not able to do what they want to do with the data inside your application.
So that's why we add this kind of business intelligence tool, or Excel feel, to AG Grid, to empower these users to be able to do all of these things.
I would like think, in a few more years, we've made AG Grid so powerful, that will be a real bona fide option for our customers to say, you know what, we can stop people exporting into Excel now, or we don't need to use these other BI tools, because we can do all of this inside of AG Grid,
Handling Multiple Framework Integration
We don't want to write any product for each framework. It just completely spreads your effort as a dev team, and then it's no longer one product and the products become out of sync. So you know, some features on one, but not in another, it's just inevitable that they don't get maintained equally, or at the same speed.
We were always a gnostic codebase, right from the start. So the first approach we went for was wrapping the grid. We would just put this thin layer over AG Grid, which was an interface into what was expected from a framework, so we'd have an Angular or react component, which would then wrap AG Grid.
What I just described is common, there's a lot of components out there who will support Angular, React and Vue but just having a thin wrapper around them.
Where we go further, is then we allow you to customise the grid using the framework of your choice. So if you're using, for example, Angular, and you're using AG Grid Angular, you can provide us with a Angular component, which we will then insert into the cells for you so that you can customise the contents of the cell using an Angular component.
And how we do that?
Well, what we have to do is engage with an API into Angular, which you would typically use if you're just a normal developer, where we could take control of the instantiation of Angular components and inject them into the DOM in these positions.
So if you're an Angular developer, the grid feels like an Angular grid, and then you can customise the Web Components written in Angular. So as far as you're concerned, it's Angular from top to bottom. And it's not just the cells, you can customise filters and editors, all those types of things.
We did that with Angular, and Vue and React.
It worked very well up untill about six months ago, where the problem that creates is sometimes the developer knows or feels that they're not working with a framework from top to bottom, and that happened in React.
The React rendering engine is quite different to the other rendering engines. It's got the virtual DOM and the rendering cycle is quite unique to React. It kind of comes with its own patterns, and react developers didn't really like how AG Grid was was integrating with React. So we started a project about six months ago that is almost complete, and we separated out the rendering side of the grid to a Functional side of the grid and that allowed the rendering piece to be plugged with a React rendering piece.
Now when you use react for AG Grid, it'll actually do all of the rendering in React all the way down to the component that you provided being the cell component. And so that means on the React side, if you're using AG Grid in react, and you open up the React developer tools, you'll be able to see your react component hierarchy, all the way down through the grid, because it's now 100% React.
About 5% of our code is front end, because I'm talking about low level DOM operations, set this class, set it with insert, this DIV, all those low level stuff. The rest of the 95% of the code, which isn't interacting with the DOM is shared between the React version and the other versions as well. The features are actually identical between the different versions.
So it's just the rendering side, we are managing two rendering engines, but most of the grid, which both of these rendering engines require is shared code.
And it's designed such that it's all based on interfaces with TypeScript. If our core part of the grid requires the rendering engine to do something we amend a TypeScript interface with the methods that way, and then if that's not implemented on either the render engines it won't compile. So in the future, if we introduce a new feature into our core, it'll do whatever it needs on the interface for the rendering engine to do, which will then force us to implement this feature on both rendering engines, therefore forcing us to keep everything in sync, and the feature list and even the end result, which I think is pretty cool.
Watch an overview video of the React UI Rendering Engine on YouTube