Building a Redis clone in Go - Part 6 (DEL, EXISTS, KEYS, & Concurrency)

Okay, guys, welcome back to another video in this Go Redis clone series. In the last part, we built the RDB data persistence method over here, which basically is going to store your database in the form of bytes on disk, right? And to implement this, we use Go's native serialization format called Gob, right? And we use that to basically create and synchronize the RDB files, right? So that's what we did in the last part of this series. In this video, we're going to build a couple of new commands: the DELETE command, the EXISTS command, and the KEYS command. Now, before we get into any of those things, I want to do some housekeeping, I guess, and go over to main.go. You'll see that, how do I show this to you? Let me open up the database file over here. You'll see that we're using a mutex over here, right? And we're using this in order to make the server concurrent, right? But is the server really concurrent right now? I mentioned this in a previous video as well, that we're going to fix this in a later part. But let me just show you, and let me stop the actual Redis server before I run this. But basically, if I run the server right now, right, and I run the client, right, it says connection accepted, right? Which is fine. But if I close the connection on the client side, the server not only crashes, but it can't even accept any future connections, right? So how is it actually concurrent if it can't even accept more than one connection, right? So we need to fix that. I said that we were going to fix it in a later part of the video, and this is that part. So let's fix that first, and then we can implement those commands: DELETE, KEYS, EXISTS, everything, right? So first of all, what we can do is we can move the connection accepting part into an infinite loop over here, right? Let's do this, all right? And let's remove this defer because we're not going to need it anymore. Let's create a wait group over here. This is what we're going to use to manage each of these connections, all right? Let's say wait group dot add one, and then let's run a goroutine, all right? Let's run this over here. What this is going to do is it's basically going to call the handle connection function, which we still need to build right after this, and then it will run the wait group dot done method. And then finally, down here, we can run wait group dot wait, so wait for all of the connections to close so we can close the server. Now let's implement this handle connection function. This is going to take the connection, obviously, and it's also going to take the state, the app state, so a pointer to the app state. And you can basically just move the entire for loop over here, right? The one that handles the connection. Move this part down here. And let's do a bit more cleaning up over here first. You can see that this read array method is going to return an error, so let's handle that as well, right? If error does not equal to nil, then we can print that error over here and then just break out of the for loop and close the connection, right? Because if this read array method returns an error, that means that the client has sent an invalid message, and we can just break out of the for loop in that case. And by the way, we can also do something like remove this function call from over here and run it directly in the if statement right here, right? You can do something like that as well. Now, that's all we need to do. But let's also add a bunch of logs over here just to make this a bit more professional, I guess. Let's say, "Accepted new connection" and print the connection's address as a string. Let's do the same thing down here and say, "Connection closed" and then the address of the connection. Now, with that done, we can pass in these arguments over here in the handle connection function, and now let's just see if all of this works. All right, so let's clear everything over here. Let's run the server. It's listening on that port. Let's run the Redis CLI over here. Okay, and you can see that it accepted my connection over here, right? Perfect. And of course, I can try getting and setting values as well. Let's open up another terminal and try creating another connection. Okay, so that didn't work, probably because I forgot to move this wait group dot wait outside the for loop. So let me just show you what's going on over here. It's accepting a connection. It's running a goroutine to handle that connection. And then it's waiting for that connection to close, right? We don't need to wait for it over here. We should just move this outside of the wait group. And we should move this actual wait group declaration outside the for loop as well. And now let's try this again. So let's close everything down. Let's run the server again. There we go. Let's run a client. It accepted my connection. Let's run a second client over here, and it accepted my connection over here as well. And I can use any of these clients to interact with the database. So let's try changing the name on this client, right? And then let's get it on the other client, and you can see that the change is reflected over here as well. So you can now accept multiple connections from anywhere in the world, I guess, and interact with this database. So let's close the client. It's gonna say, "Connection closed," and it's going to gracefully handle the entire thing. Let's close the other client as well. It says, "Connection closed," and that's basically it. We are done with this. Awesome. And now we can shut down the server as well. So no more errors, no more server crashes, nothing, right? This is a fully concurrent server, unlike what we had before. Now, with all of that done, and I guess we don't need this wait function call over here, but I'm just going to leave it there. It really doesn't matter, right? So now we have a fully concurrency-supported server. Now we can finally move on to implementing those commands that I said we would implement. So let's go over to handlers, right, and set up the delete command. I'm going to say delete equals the delete function, and let's implement that down here. Let's say function delete. This is going to take a value and a state, just like all of our other handlers, and will return a value as well, which will be the reply we send back to the client. Now the delete command works like this. You basically send the delete command along with a bunch of keys, right? A list of keys that you want to delete from the database, and the delete command will return, right, the number of keys that have been deleted, right? Now, emphasis on the number of keys that have been deleted, right? It's not going to return the number of keys that you tried to delete, right? It's going to return the number of keys that have actually been deleted from the database. So if we're deleting these three keys, right? Key one, key two, and key three. Let's say key two doesn't exist in the database, right? So it's not going to get deleted because it doesn't exist, which means we're only going to delete keys one and three, and we're going to return two, which is the number of keys deleted from the database. Not the number of keys we tried to delete, but the number of keys that were actually deleted. All right, so that's how the delete command works. Now, let's implement this. First of all, let's get the arguments from the client. This is going to be everything after the first element, which will be the command, obviously. And now let's create an integer value over here. This is going to count the number of keys that have been deleted, and we will return this in the reply. And now, obviously, before we make any changes to the database, we should lock it as well, right? Lock the mutex, and I'm locking it for writing as well because delete is kind of a write operation. We're deleting values from the database. We're not reading values from the database, right? So lock and unlock the writer. And then in here, let's loop over every single argument in this args list. Let's first of all see if it even exists in the database, right? So let's say DB dot store and arg dot bulk. And if it does exist, right, then we should delete it first of all, and you can say write. So we can pass in the map first of all, which will be the db.store, and then the key that we want to delete, which will be arg.bole. If this key actually existed in the database, then we can increment the n over here, and that's about it. Yeah, so unlock the writer after that because we don't need to modify the database after this, and return a value. Now, before we return this, the delete command expects a return value in an integer data type, which we haven't implemented as of right now. So let's do that. First of all, let's go over to the value file over here and implement integer, right? So create a new value type, call it integer, and this will have the symbol of a colon. All right, that's done. Let's go over to writer over here and in the deserialize method down here, let's actually return the integer value as well. Let's say reply equals fmt.Sprintf, first the type, which will be the integer type, and then the actual integer. All right, v dot, let's say num. We need to create this field. We haven't created it yet, so let's go over to value again, and over here, let's create a num field, which will be an integer. That's all. And then finally, let's add the new line characters to tell Redis that you can stop scanning the message after this. All right, now hopefully we did all of that correctly. Let's try returning this. We're going to return an integer, and the integer will be the n field that we declared up here. Now let's try to actually use this. All right, so let's run a server. Let me close one of these terminals. Let's run the client. Now let's try to get the name, and it's returning my name from the, uh, I guess the AOF records over here. Let's try deleting that name, and you can see it returned an integer with the value of one, which means that one of the keys was deleted, right? So let's try getting it again, and it's nil now because it got deleted from the database. Let's try deleting it again, and this time it returns zero. Why? Because no key was actually deleted. This name key was already deleted. It didn't exist in the database anymore, and so we got zero because no key was actually deleted this time. So that's how that works. Let's try getting the age as well, and it's set to 20 right now. Let's try setting the name again. All right, now let's try getting the name. So you can see that both the age and the name keys are right, are set right now. Let's try deleting both of them at the same time, and let's also try deleting a random key over here, right? It's going to return two because name and age are valid keys in the database, right? And both of them got deleted. This random key doesn't actually exist in the database, so it didn't get deleted. And that's it, right? So it's safe to run delete on any key, right, that you don't know if it exists in the database or not, right? It's safe to run delete. It's not going to return an error. It's just going to return how many keys were actually deleted, right? So that's all we need to do for the delete command. Next, let's implement the exists command as well. So go over here in the handlers list and set up the exists command over here. Let's also implement it down here. Let's say exists, pass in the value and the app state, and return a value as well. Now this is very similar to the delete command, but it's only going to lock the reader, not the writer, because in exists we don't really modify any of the keys, right? We just check if the given keys exist. But it is very similar to the delete command in that it's going to take a list of arguments, right? And it's going to have another integer as the return value. Let's lock the readers and unlock them down here. All right. And as the reply, for the reply, we're going to return a value of type integer with the num keyword set to n, right? Now in here, let's loop over all of the arguments. Let's see if they exist in the database, right? And if they do, then let's return or let's increment the n variable over here. And that's all we need to do, right? So let's run the server again. There we go. Let's run a client. If I try to get name, you can see that it does exist, right? So if I say exists name, it gives me the value of one. If I do exists name and age, right, it should give me two. And there we go, because both of these exist. If I do something like some random key that doesn't exist, it's not going to return five. It's going to return two because both of these exist, but this key three, four, and five, they don't exist in the database, right? So that's the exist command. You can see these are very simple commands to implement over here. Finally, let's do the keys command over here, and let's say it's going to take the value again and the app state and return a value as the reply to the client. Let me close down the server and the client over here. So the keys command basically will take only one argument, right? So first, let's just grab the list of arguments over here. And let's make sure that it's only one argument. If not, let's return a value over here, which will be of type error, and the error message can be something like, "Error: invalid number of arguments for the keys command." All right, now the keys command can only accept one argument, and that argument will be the pattern that we need to match against the keys in the database. So let's grab that argument as well, and the bulk field inside that argument, and let's lock the database for reading because that's what we're doing in this command. We don't need to write anything. We just need to read, right? Now in here, how the keys command works is it basically grabs it, it basically gets a pattern right over here, and then we just match that pattern against all of the keys in our database. And for every key that matches that pattern, we return it to the client. So basically, if I have something like, let's say, if I have something like name and age in my database, I can run something like keys name to get the name key back. I can also run something like NA and then a wildcard character, which basically means any number of characters after this, right? And it will still return the name field, right? The name key. So basically, what this does is it uses glob-style pattern matching, right? And it uses that to return the number of keys that match that pattern. It's going to make a lot more sense once I actually implement this and show it to you. All right, so let's do that. Now, mind you, this is kind of an expensive command to run in that it takes a lot of, like, server resources depending on how many keys you have. So, well, we don't need to worry about that really because this is like a toy project. But in production settings, what you use is, instead of using the keys command, you use the scan command. I might implement that as well in a future part of this video, but we'll see. So let's create a matches list. This will be a string list, right? And then let's loop over every single key in the database store field over here. Now, there's a bunch of different rules for glob-style pattern matching, and instead of implementing all of those ourselves, which is going to take so much more code than this, we can use the filepath package. So we can do something like filepath.Match, right? And then we just pass in the pattern in here, and then just pass in the key name in here. This is going to return two fields, the matched field and the error field. Let's handle the error first. Let's log the error. Let's say, "Error matching keys." Let's print the pattern over here, and let's also print the key that we try to match over here. So we're printing all of the relevant data alongside the error, and this is just going to, like, make it a lot easier for us to debug this whole thing if we need to. And let's also print the error as well, I guess. I believe you do that with the percentage V, but if not, you can always fix that later, and return after that. After printing the error, or not return, actually, but just continue to the next key in the database. And if this pattern matches, we should append it to the matches list over here. Append the key over here. And I forgot to add the range keyword up here in the loop. Please don't do that. Make sure to add the range keyword so this works. Now that all we need to do to grab the matches. Finally, let's create our reply. This needs to be a value of type array because that's what the Redis client expects as the return value in this keys command, right? So let's create that, and then let's loop over all of the matches, right? And let's append The match to the reply dot array field by doing append reply dot array. And then over here, let's append a value of type bulk, and the bulk will be set to the key name, all right, which will be this m value in this matches list. So basically, what we're doing is we're returning an array of key names, right? And then we just return that over here as a reference and remove this colon over here. We're not declaring a new variable; we're just reassigning to an existing variable. So make sure to do that as well. And that's basically all we need to do now. Let's add this up in the handlers list over here, right? Always make sure to do that. There we go. Save everything, and let's try running this now. So let's run the server first of all, then let's run the client, right? And let me just try setting a bunch of new keys. So let's say key one equals x, set key two equals y, and set key 3 equals z, right? So all of these keys have been set now. Let's clear everything and try to run the keys command and say something like key and then a wild card, right? Which basically means any number of characters after this, and it's going to return all three of these keys, right? I can also do something like if the h key exists, right? And it does. I can do something like a g and then a question mark, which basically means only one character after this. So any given character after this, and if it matches the keys in the database, and it does, it's going to return the key in a list over here, right? So basically, in this glob style pattern matching, you can use symbols like question mark, which basically means one character. You can use wild card, which basically means any number of characters after this, or you can just type the complete name of the key, and that works as well, right? You can even do something like something more complicated, like something like, I don't know, n wild card and then e, which basically means the n character and then the wild card, which basically means any number of characters after this, and then finally at the end, an e character. And this is still going to match the name key in our database. So you can see there's a, there's a bunch of different rules in this, in this glob style pattern matching, right? This is just the, like, kind of a summary that I'm giving you over here, but there are a couple more tricks you can use to write a bunch more complex patterns to match, right? And instead of implementing all of them by ourselves, we just use the file filepad.match function to do it for us, right? And here in the documentation, they're showing you a bunch more characters you can use. You can use the wild card, you can use the question mark, you can use these other symbols as well. You can go down this rabbit hole and study this more if you want to, but this isn't really relevant to the project, so I'm going to skip that. But yeah, basically, that's the keys command. Yeah, there we go. So that's the end of this part of the series. Let me just go over everything again. We, in this part, we finished implementing the concurrency part of this server, right? We fixed all of the server crashes. We set up the server to accept and interact with multiple clients at the same time over here, right, by using a go routine. There we go, by using a go routine and this handle connection function and everything. And then in the handlers list over here, we created these three new commands. We created the keys command, we created the exists command, and we created the delete command, right? So this server is coming along very nicely. This project is coming along very nicely. We're very far into this, right? In the next part, I'm not sure what I'm going to do in the next part yet, but I am planning to implement the save and the bg save commands, right? So let me just give you a quick overview of what these commands do. Remember that the, when I was talking about the data persistence methods, right, back in a previous video, I mentioned that the, there are two ways to save an rdb snapshot. One is to actually have this save keyword in the configuration file and have a bunch of like conditions after that, and this is basically the automatic way to save an rdb snapshot. But the save command over here, this is a manual way to save an rdb snapshot, so I could run something like, I don't know, the save command over here, and this would create an rdb snapshot manually, right then and there, right? And save is also a very expensive operation to run because you're basically looping over all of the, like, the entire database store, right? And if you have something like thousands or millions of keys in your database, which is pretty common, I guess, for really large applications, really like large enterprise servers, right, then it can hang your database for a very long time, right? And that is why we can also use bg save, which basically just stands for background save, and what this does is it doesn't block the actual database, and it just saves the rdb snapshot in the background, right? So I'm going to try to implement both of these in the next part of the video. So stay tuned for that. Thank you for watching this part. I hope you learned something. I hope you're building this project alongside me and learning all sorts of new things just as I am. Yeah, thank you for watching. Like, subscribe, share the video, comment if you have any questions or something or if you just want to say hi or whatever. And I will see you in the next video. Thank you for watching this one. Stay tuned for the next part. I'll see you in the next one. Bye-bye.