Creating API Rate limiter in Nodejs using express and Redis

If you are building API Key based service, API Rate limiting is one the most important feature. This feature is useful when you are in Banking domain or a matter of fact in any domain, where you want to control the number of requests sent by the API consumer. As the page title reads in this article, we will create api rate limiter in nodejs using express and redis.

1. Why Rate limiting?

=> Let’s take the example of API service that you are creating which needs API KEY in order to be consumed. Now you want to restrict the user based on two conditions,

  1. Users having API key can access your API 10 times.
  2. And Users who don’t have API key, can’t access your API service more than 5 times.

=> So in such situations API Rate limiting comes very handy and implementing API Rate limiting in Nodejs will piece of cake for you after reading this article. Now in this article, we will apply rate limiting based on few criteria and some request verb as shown below,

  1. Rate limiting on User IP address.
  2. Rate limiting on GET requests.
  3. Rate limiting on POST requests.
  4. We will apply the Rate limiting on API KEY as well.

So let’s start.

     Download

 




 

2. Creating a new Nodejs project

1. Let’s start off by creating a new Nodejs project by using npm init command. This command will create a new package.json file.

2. After that copy the dependencies from below package.json and paste it in your file and run npm install.

Below is my package.json file for this application.

package.json:

3. Project structure and Files

=> If you have followed my previous Nodejs articles, you might familiar with folder structure I create. I try to keep the file structure as realistic as they can be so that you can easily integrate them into your projects. Also, we follow Object-oriented style, so that makes our code more elegant and clean.
=> The below images represents the file structure of our project. There are four files that we are going to use, let’s understand the purpose of each file,

1. config.js : In this File, you will register all your Rate Limiters, that will be used by routes as a middleware.
2. redis-database.js : The name of the file screams its purpose, in this file we will connect the Redis database.
3. routes.js : In this file, we will register the routes of the application.
4. server.js : In this file, you will setup your NodeJs server and application configurations.

api rate limiter nodejs using express redis folder structure

4. Setting Up the Nodejs server and connecting Redis database.

=> So here we will first set up the nodejs and then we will initialize the routes of the application. As I said in the last section we will do our setup in the server.js file.

=> So open the server.js file and add the below code, the below code is very straightforward and easy to understand so I will leave to up to you. Let me know in the comment box if you don’t understand something.

server.js:

=> To make this project work, you will require redis database, now why you need a database in the first place ? we will get to this question the next section, for now just be with me.

=> So to connect our Nodejs application with the redis database, we will use redis nodejs module. Open the redis-database.js file and write down the below code,

redis-database.js:

4. Creating API rate limiter using express and Redis

Now the question is, why Redis? Why not any other database? Well, Redis is very fast as compared to storing and pulling out data from the database. So Redis becomes the obvious choice for us. Now the second step is to find any good NPM module, which can help us to implement it. Luckily we have got a good NPM module it’s called express-limiter which will help us to implement rate limiting.

1. Rate liming using the IP address

=> Let’s first start adding the rate limiting on IP address. The first thing you will do is, add a route in routes.js file on which you want to set API Rate Limiter.

=> Later, you will add the rate limiter configurations in the config.js file. Open the routes.js and add the below code,

routes.js:

Explanation:

  1. First, you would import the config.js file, which will basically contain all your Rate Limiter configurations.
  2. The next thing would be creating the object of RateLimiter class by passing the instance of the express app. By creating an object of the RateLimiter class, you will get access to methods of that class.
  3. Later in appRoutes() method, you will register all your routes. As of now, we have only one route i.e. /users.
  4. If you notice carefully, we have passed the second parameter which is nothing but a route middleware. This middleware is a method of RateLimter class.
  5. usingRemoteAddress() method will return the rate limiter configuration.

Now let’s open the config.js file and write down the rate Limiter configuration for the /users routes.

config.js:

Explanation:

  1. The first line of the file will give us the Redis database connection and we have already written the code to connect redis database in above section.
  2. Then in the constructor method, we have initialized the express-limiter module by using an instance of Express app and Redis connection client.
  3. And at the end, we have usingRemoteAddress() method which returns the limiter function. The Limiter() function expects few parameters, as listed here.
  4. Here lookup: ['connection.remoteAddress'], will apply the rate limiter based on Remote IP address on specified Path and Verb.

2. Rate liming based GET and POST request Parameter

=> In the last section we saw how to implement the basic Rate limiter using IP address. So the implementation will be an almost identical only thing will get changed is that, we will add another the Limiter function in the config.js file and use it as a route middleware.

=> Now open the routes.js file add the below route, you know where to add right!

route.js:

=> The step would be to add the Middleware inside the config.js file. The middleware is a method defined inside the config.js file, which will return the Limiter() function.

Open the config.js and add the below method into it.

config.js:

Explanation:

  1. Now inside the this.limiter() function you have to pass an object having some specific keys which I have listed below,
    • The path should be same as the route, on which you intend to apply the Rate limiter.
    • The method should be get/post or any other mehtod based on the route.
    • lookup: ['params.id'], will check for the id parameter. Basically, it will check for the id object inside params object which falls under request object (I hope you got this point If you don’t, comment in below comment box).
    • total indicates the number of requests allowed.
    • expire is time after which user can use the particular route.
    • onRateLimited() function will return the response back to client.
  2. And at the end, you to return the this.limiter() function.

3. Rate liming based on API KEY

=> Suppose you are creating an API service and you withstand the situation where you have to allow a user having API KEY can send n number of requests to the server. And users who don’t have API KEY they can send only 100 requests per day.

=> Now, in such cases Rate limiting based on API KEY becomes one of the most important and demanded features if you are creating an API service. Here we will implement the same feature, just like we did in last two example.

Example:

  1. For example, I have URL localhost:4000/details/API_KEY, and the user will send a request to this URL in order to access some data.
    Note, This can be GET or POST request
  2. Now users having API_KEY can send 10 requests within an hour and the users who don’t have API_KEY can send only 5 requests within an hour.
  3. To check if the user has valid API key we can make a database query and cross verify if the API key provided by the user is valid or invalid.

=> Now let’s start the implementation, open the routes.js and add the below route,

routes.js:

=> Now let’s add the Rate Limiter in route, as of now I am not querying from tha database to verify the API Key. But in a real-world application, it is a better choice to do.

Open the config.js and add the below code, and the below method into it.

config.js:

Explanation:

In the checkApiKey() method, we are returning the limiter() function, which will basically contain the code for the Rate Limiter. As you know the rate limiter function expects an object with some key and values associated with it. Let’s go through each key and values are given to those keys.

  • path/details/ will the route where you want to apply the Rate limiter.
  • method: Whatever your route VERB is.
  • lookup: This key can take three types of arguments, here we will pass function. In this function will basically check the if the API Key is valid or not. This function has four parameters listed below,
    • request => This your usual route request object.
    • response => As the name suggests this is your route response object.
    • opts => This is an important parameter, since this where you will apply which type of Rate limiting should take place.
    • next=> This is for middleware (a Nodejs thing).
  • Here we are using isValidApiKey() to perform the API key check, this method returns a Promise with a boolean value. In this method, you can execute database query check.

Final Thoughts

As you know, nowadays Websites are becoming more API oriented. Thus Rate Liming becomes of those features that as a developer you must have under your umbrella. Here in the article, we studied three types of most commonly used ways to implement Rate Limiting in any API service. You would find me using one of these are a Rate Limiter every day, if you think I forgot to include any popular Rate Limiter do let me know in the below comment box.