Building a Redis clone in Go - Part 1 (TCP server, RESP protocol)

Alright guys, welcome back to another video. In this video, we're going to build a Redis clone inside Golang. So Redis, you probably already know about that. It's kind of like an in-memory database, a database that you store in memory. It's basically a key-value store, and we're going to build a clone of it in Go. So first of all, let me just create a project folder for this. I'll just call this Go Redis Video, and I'll cd into this and open up my VS Code window inside this folder. So first, what we need to do is install Redis. Alright, so I already have it installed, so let me just uninstall it just so I can show you what the process is like, right? So let's uninstall this whole thing, and now I'm just going to open up the Redis docs and install it all over again. Alright, so obviously, if you're on Windows, you go to the Windows section over here. If you're on Mac, you go to the Mac section. But since I'm on Linux on WSL, I'm going to follow these instructions. Alright, so all you have to do is just copy-paste these commands, and you can install Redis. I'm just going to do all of this real quick, and yes, override it, because I already did this before. There we go. Super simple, copy and paste, all right? There we go. So you've installed Redis now, all right? Make sure you do this as well so you can follow along. Now, what we need to do next is, so Redis, when you install Redis, it comes alongside a client and a server, all right? So what we're going to do is use the Redis client that we get from this installation and create our own Redis server, all right? So our server is going to be able to communicate with any Redis client, all right? Because we're going to satisfy the required protocols and everything. So what you want to do first is stop the actual Redis server that's running right now. This is the official server that comes along with this Redis installation, all right? We want to stop this because we're going to create our own Redis server in place of this. So make sure you stop this. And if I try out Redis CLI right now, this is the client, it's not going to work, right? Because the server is down. So now we can actually start creating a server. First of all, let's just create a main.go file over here. I'll call this package main. I'll add a main function, and what I'm going to do right now is just build a basic TCP listener, all right? So using the net package, I'm going to say net.Listen on the TCP network on localhost port 6379. So this is the default port where the Redis server is hosted. All right, so we're just going to copy this one. Make sure we handle the error. If we can't connect to this port, then that means we can't even run anything, right? So let's just exit the program right there with an error message: "Cannot listen on port 6379." There we go. Now we have a basic TCP listener on this port, right? So now what we need to do is accept connections over here. So let's say con, which is connection, and error equals l.accept. L is the listener, obviously. What this is going to do is it's going to block the code until a connection is opened on this port. All right? Again, handle the error as well. Let's print the error and exit the program over here, all right, if there's an error. Now we have a connection. What we can do now is we can read everything in here. I'm just going to build a very basic Redis server for now, right? It's not going to do anything because I just want to get the basics out of the way first, all right? So what we need to do now is create a small buffer. I'm just going to make it a thousand and twenty-four bytes, all right? So one whole KB. We pass that in over here. As you can see, this function requires a byte slice, so we pass in a buffer over there, and then we can actually just print what we get from this buffer, convert it into a string, obviously, and then just print it. And now we can write back to the client a message that it will understand. So for now, just copy and paste this. I will explain what's going on, but for now, just copy and paste: plus OK backslash r backslash n, all right? And obviously, it's giving me an error because this needs to be in bytes, so convert this to a byte slice like this. That's it, all right? Up here, I forgot to do this, but make sure you add a defer to close this listener when you exit the program. Same for this connection as well. There we go, just so we're freeing up the port and freeing up the resources that we're using, right? So let's actually play around with this now. Let's see if we actually made this properly. I'm going to run. I forgot to create a mod file. Okay, so I forgot to create a mod file, so let's do that first. We'll call this Go Redis Video. There we go. Now let's run this again. Okay, so it's listening on that port now, on 6379. So let me open up Redis CLI, and there we go. This time, you see we don't get the connection refused error, right? The client can actually connect to the server properly, and you can see over here what happened is it sent, so the client sent a bunch of commands, and we printed all of them over here. Now, this obviously doesn't make much sense right now because this is in RESP format, so RESP stands for Redis Serialization Protocol, all right? All of the messages that the Redis client sends and the Redis server sends are going to be in this RESP format, all right? Redis Serialization Protocol. What we need to do is we need to parse this protocol and convert it into a format That we can understand, and then we just need to store data, retrieve data, and do all of those basic things, right? So instead of exiting the program over here, what we can do is have an infinite loop. Let's just say there we go. So by doing this, at least the server won't, like, shut down after just one message. All right, so let's run the client again and then just try sending a bunch of commands, right? Like, get name, set name to Hassan, that's my name, get name again. And you can see in every single instance, it's just sending this OK message that we're sending over here, right? It's not sending anything else because that's all we've coded for now. And you can see every time I send a message over here in this client, the server prints all of these, like, weird letters and everything. All right, so this is what we actually need to understand now because this is the RESP protocol, and this is what we need to implement if we want to create a Redis server. So let's go through this. All right, you can see that every single one of these messages starts with a star symbol. The star symbol means array, all right? So every single command that the client is going to send to the server is going to be an array that is filled with a bunch of commands, all right? So let's create our first data type up here. I'll create a type called value type, which is going to be a string, all right? And then I'll create a bunch of constants. All right, so the first one is going to be array, which is going to be of type star, all right? This little star you see over here. Now, after the star, this two means how many elements are in this array, all right? We don't need to worry about that right now. Let's just keep defining our data types. After that, you see this dollar sign. The dollar sign means a bulk string, a multiple-word string, basically, and every single message that the client sends to the server is going to be an array that is filled with a bunch of bulk strings. So let's handle that as well. Let's create a data type for bulk as well. This is going to be what's going to be sent in the command. And let's also create a value type for string. Now, string is represented by this plus sign over here. All right, you can see I'm sending plus OK. The letter is OK, and then backslash R, backslash N. All right, so this plus sign means a string, simply a string of one word. So let's handle that as well. The symbol for this is obviously going to be the plus sign. There we go. So now we have a bunch of value types. Now let's also create a struct called value, which we will use to actually convert this whole message to a format that we can understand, all right? So I'll create a struct. I'll give it a type, which can just be value type. I'll also create fields for bulk, which is just going to be a string, another field for string, and also a field for array, which is just going to be a list of value types. There we go. All right, so now we have our value struct. All right, now we need to actually convert these little messages over here into this value struct. So I mentioned before that every single message that the client is sending is going to be an array, right? So now let's create a method to read those arrays. I'm going to create a pointer receiver for this new value struct that we created, and I'll call it read array. All right, there we go. And what this is going to do is it's going to take a reader. So reader will just be an io.Reader interface. And I'm going to go down here, and instead of reading the connection over here and printing all of this, what I can do is I can just call, or I can create a new value and give it the type of array because every single message, again, every single message that the client is sending is going to be an array, all right? So we can just hard-code that over here. Then we can just do read array and pass in the connection because the connection satisfies the io.Reader interface because it implements the read method, all right? So it satisfies the reader interface, so we can just pass in the connection here directly. Now we have the connection over here. Now we can read the bytes that the client is sending and then just parse it and convert it into a value, all right? So let's do that first. Let's create a buffer that can store four bytes and then read those bytes, all right? So this is going to convert this first line over here, all right? The star and the two. Let me just maybe I can keep this over here just so it's easier to see. There we go. So to read this first line, we need to store at least two bytes. So let's read this. Now, the data that we want to fetch from this buffer is in the second element. This is going to be the actual length of the element. So let me just print one of the messages again, I guess. Or let me just finish this and I'll explain. Not making much sense right now. So we create a buffer that can store about four bytes, right? Exactly four bytes, and then we read those four bytes from the connection, all right? Now, there's no need to print this right now, I guess. But what we can do is we can now get the length of the array, right? So to get the array length, we'll just say array length and error equals strconv.Atoi, which is ASCII to integer, I guess that's what it stands for. And the string over here is going to be string, and then the second element in the buffer. This is the array length, all right? So you saw that the first line over there was something like star two, right? And then just a new line, right? Like this. This is why we need to store it in four bytes, right? So one byte, two bytes, three and four. There we go, right? And the length of the array is in this second byte. So that's what we fetch over here. We convert it to a string from bytes, and then we convert that string into an integer. So make sure you handle the error as well. For now, I'm just going to print the error, right? Because I don't really care. There we go. And then let's just use this array length to then parse the rest of the array, all right? So what we can do is say `for range array length`. Now, again, I mentioned that the messages that the client sends are just an array, right? Now, that array, and I said this before as well, the array that the message that the client sends is just going to be a bunch of bulk strings, right? So now we need to create a method to read those bulk strings. So what I'm going to do is I'm going to say bulk equals V dot read bulk. And we need to actually create this method, obviously, and I pass in the connection again as the reader over here. So now, obviously, we need to implement this read bulk method. But we're gonna receive the bulk value over here, and then I append it to this array in the original value, all right? Pass in bulk over here, and then you can just return this value as such. Actually, we don't even need to return it because we're modifying the pointer directly, right? So we don't need to return it. The changes are going to reflect in that value anyway. What we can do now is we can create another pointer receiver, and this is going to be read bulk. It's going to take another reader as well, and this is actually going to return a value, all right? And we're going to store that value inside this array, all right? So let's do that next. So this is going to be a bit similar to the array. First, we need to actually create a buffer to store four bytes, and again, we're just going to read those bytes from the connection. Then we're going to get the length of the bulk string, all right? So this is again just going to be the second element. I'll just call this n, and again, we'll just convert this to an integer. The second element. There we go. Make sure we handle the error again. Normally, here you would return the error so that the caller can handle it in whatever way it chooses to, but for now, since this is not a serious program, since we're just trying to get this over with, I'll just print the error and return, all right? Return nil or just return an empty value. There we go. And now we can actually use this n value to read the bytes. So let's create another buffer for this bulk string. This is gonna have a capacity to read n bytes, which is the length of the string, and then two more, which will be the new line characters r and n, right? So we need to make sure that we read those as well. So we add a capacity for two extra bytes. Then we just read that, and then we have the bulk string. Now we need to convert this to a bulk string, and we also need to just remove those last two new line characters, right? So for that, we can just do this, or not this, but more like this, right? So this is going to remove the last two elements and just give us the string itself, and then we can just create a value, right? Give it the type of bulk and assign the bulk value over here. Awesome. There we go. So now we have all this done. What we can do now is we can read the array and then just, I guess, print it and just see what we get, right? So I'll run the server again, and okay, it can't listen on 6270. Let me stop the actual Redis server. I did this before as well, but my computer crashed, and when I restarted it, I guess the Redis server just restarted on auto start or something. Who cares? Make sure that the Redis server is stopped, and then launch your own server. Now it's running. Now let's run the CLI. There we go. I'll pass in a bunch of commands as such, and you can see now we're getting the actual commands over here, right? And we can just read these. We have them stored in like a format that we can understand in this value struct. Okay, so now you can see that instead of having those weird strings in RESP format, right, Redis Serialization Protocol, instead of that, now we can actually parse those strings into this value struct that we can actually understand, right? So now that we've gotten this far, I'm going to end the video. In the next part, I'm going to build commands like GET, SET, EXISTS, DELETE, and other commands like that, right? So the actual key-value store, the actual database stuff, storing and retrieving data, that's what we're going to build in the next part. In this first part of the video, I just introduced you to Redis and the Redis Serialization Protocol, how it looks like, how it works. And we just built a bunch of methods to actually parse that RESP message from the client and just understand it a bit more, right? So this is good progress for part one. In the next part, we're going to handle all of the storing and retrieving of data, right? So stay tuned for that. Thank you for watching. Like, subscribe, all of those wonderful things. Thank you for watching, and bye-bye.