Auto-generated transcript Okay guys welcome back to another video in this go Redis clone series In the last part we built the RDB data persistence method over here Which basically is going to store your database in the form of bytes on disk, right? And to implement this we use the goes native serialization format called gob right and we use that to basically create and synchronize the RDB files right so that's what we did it did in the last part of this series in this video we're going to build a couple of new commands the delete command the exists command and the keys command now before we get into any of those things I want to do some housekeeping I guess and And go over to main.go. You'll see that... How do I show this to you? Let me open up the database file over here. You'll see that we're using a mutex over here, right? And we're using this in order to make the server concurrent, right? But is the server really concurrent right now? I mentioned this in a previous video as well, that we're going to fix this in a later part. But let me just show you and let me stop the actual Redis server. before I run this. But basically, if I run the server right now, right, and I run the client, right, it says connection accepted, right, which is fine. But if I close the connection on the client side, the server not only crashes, but it can't even accept any future connections, right? So how is it actually concurrent if it can't even accept more than one connection, right? So we need to fix that. I said that we were going to fix it in a later part of the video and this is that part. So let's fix that first and then we can implement those commands. Delete, keys, exists, everything, right? So first of all, what we can do is we can move the connection accepting part into an infinite loop over here, right? Let's do this, all right? And let's remove this defer because we're not going to need it anymore. Let's create a weight group over here. this is what we're going to use to manage each of these connections all right let's say weight group dot add one and then let's run a go routine all right let's run this over here what this is going to do is it's basically going to it's basically going to call the handle connection function which we still need to build right after this and then it will run the weight group dot done method and then finally down here we can run weight group dot weight so wait for all of the connections to close so we can close the server now let's implement this handle connection function this is going to take the connection obviously and it's also going to take the state the app state so a pointer to the app state and you can basically just move the entire for loop over here right the one that handles the connection move this part down here and let's do a bit more cleaning up over here first you can see that this read array method is going to return an error so let's handle that as well right if error does not equal to nil then we can print that error over here and then just break out of the for loop and close the connection right because if this read array method returns an error that means that the client has sent an invalid message and we can just break out of the for loop in that case and by the way we can also do something like remove this function call from over here and run it directly in the if statement right here right we can do something like that as well now that's all we need to do but let's also add a bunch of logs over here just to make this a bit more professional I guess let's say accepted new connection and print the connections address as a string let's do the same thing down here and say connection closed and then the address of the connection. Now with that done we can pass in these arguments over here in the handle connection function and now let's just see if all of this works. All right so let's clear everything over here. Let's run the server. It's listening on that port let's run the redis cli over here okay and you can see that it accepted my connection over here right perfect and of course i can um try getting and setting values as well let's open up another terminal and try creating another connection okay so that didn't work probably because i forgot to move this weight group dot weight outside the for loop So let me just show you what's going on over here. It's accepting a connection It's running a go routine to handle that connection and there's waiting for that connection to close, right? We don't need to wait for it over here We should just move this outside of the of the weight group and we should move this actual weight group declaration Outside the for loop as well. And now let's try this again So let's close everything down Let's run the server again there we go let's run a client and accepted my connection let's run a second client over here and it accepted my connection over here as well and I can use any of these clients to interact with the database so let's try changing the name on this client right and then let's get it on the other client and you can see that the change is reflected over here as well. So you can now accept multiple connections from anywhere in the world I guess and interact with this database. So let's close the client. It's going to say connection closed and it's going to gracefully handle the entire thing. Let's close the other client as well. It says connection closed and that basically it We are done with this Awesome And now we can shut down the server as well So no more errors no more server crashes nothing right This is a fully concurrent server unlike what we had before Now with all of that done and I guess we don need this wait function call over here, but I'm just going to leave it there. It really doesn't matter, right? So now we have a fully concurrency supported server. Now we can finally move on to implementing those commands that I said we would implement. So let's go over to handlers. right and set up the delete command i'm gonna say delete equals the delete function and let's implement that down here let's say function delete this is going to take a value and a state just like all of our other handlers and we'll return a value as well which will be the reply we send back to the client. Now the delete command works like this. You basically send the delete command along with a bunch of keys, right? A list of keys that you want to delete from the database and the delete command will return, right? The number of keys that have been deleted, right? Now emphasis on the number of keys that have been deleted, right? It's not going to return the number of keys that you tried to delete, right? It's going to return the number of keys that have actually been deleted from the database. So if we're deleting these three keys, right, key 1, key 2, and key 3, let's say key 2 doesn't exist in the database, right? So it's not going to get deleted because it doesn't exist, which means we're only going to delete keys 1 and 3, and we're going to return 2, which is the number of keys deleted from the database. Not the number of keys we tried to delete, but the number of keys that were actually deleted. All right, so that's how the delete command works. now let's implement this first of all let's get the arguments from the client this is going to be everything after the first element which will be the command obviously and now let's create an integer value over here this is going to count the number of keys that have been deleted and we will return this in the reply and now obviously before we make any changes to the database we should lock it as well right lock the mutex and I'm locking it for writing as well because delete is kind of a right operation we're deleting values from the database we're not reading values from the database right so lock and unlock the writer and then in here let's loop over every single argument in this args list let's first of all see if it even exists in the database right So let's say db.store and arg.bulk. And if it does exist, right, then we should delete it, first of all. And we can say, right, so we can pass in the map, first of all, which will be the db.store. And then the key that we want to delete, which will be arg.bulk. And if this key actually existed in the database, then we can increment the n over here. and that's about it yeah so unlock the writer after that because we don't need to modify the database after this and return a value now before we return this the delete command expects a return value in an integer data type which we haven't implemented as of right now so let's do that first of all let's go over to um the value file over here and implement integer right so create a new value type call it integer and this will have the symbol of a colon all right so that's done let's go over to writer over here and in the deserialize method down here let's actually return the integer value as well let's say reply equals fmt sprint f first the type which will be the integer type and then the actual integer all right We thought let's say now we need to create this field. We haven't created it yet. So let's go over to value again and Over here. Let's create a num field which will be an integer that's all and then finally, let's add the new line characters to tell Redis that you can stop scanning the Message after this. All right. Now, hopefully we did all of that correctly. Let's try returning this we're going to return an integer and the integer will be the n field that we declared up here now let's try to actually use this alright so let's run a server let me close one of these terminals let's run the client now let's try to get the name and it's returning my name from the I guess the AOF records over here let's try deleting that name and you can see it returned an integer with the value of one which means that one of the keys was deleted right so let's try getting it again and it's nil now because it got deleted from the database let's try deleting it again and this time it returns zero why because no key was actually deleted this name key was already deleted it didn't exist in the database anymore and so we got zero because no key was actually deleted this time so that's how that works um let's try getting the age as well and it's set to 20 right now let's try setting the name again all right now let's try getting the name so you can see that both the age and the name keys are right are set right now let's try deleting both of them at the same time and let's also try deleting a random key over here right it's going to return 2 because name and age are valid keys in the database right and both of them got deleted this random key doesn't actually exist in the database so it didn't get deleted and that's it right so it's safe to run delete on any key right that you don't know if it exists in the database or not right it's safe to run delete it's not going to return an error it's just going to return how many keys were actually deleted, right? So that's all we need to do for the delete command. Next, let's implement the exists command as well So go over here in the handlers list and set up the exists command over here Let also implement it down here Let say exists Pass in the value and the app state and return a value as well Now this is very similar to the delete command, but it's only going to lock the reader not the writer because in exists We don't really Modify any of the keys right we just check if any if the given keys exist So but it is very similar to the delete command in that it's going to take a list of arguments, right? And it's going to have another integer as the return value let's lock the readers and unlock them down here all right and as the reply for the reply we're going to return a value of type integer with the num keyword set to end right now in here let's loop over all of the arguments let's see if they exist in the database right and if they do then let's return or let's increment the n variable over here and that's all we need to do right so let's run the the server again there we go let's run a client if i try to get name you can see that it does exist right so if i say exists name it gives me the value of one if i do exists name and age right it should give me two and there we go because both of these exist if I do something like some random key that doesn't exist it's not going to return five it's going to return two because both of these exist but this key three four and five they don't exist in the database right so that's the exists command you can see these are very simple commands to implement over here finally let's do the keys command over here and let's say it's going to take the value again and the app state and return a value as the reply to the to the client let me close down the server and the client over here so the keys command basically will take only one argument right so first let's just grab the list of arguments over here and let's make sure that it's only one argument if not let's return a value over here which will be of type error and the error message can be something like error invalid number of arguments for the keys command all right now the keys command can only accept one argument and that argument will be the pattern that we need to match against the keys in the database so let's grab that argument as well and the bulk field inside that argument and let's lock the database for reading because that's what we're doing in this command we don't we don't need to write anything we just need to read right now in here how the keys command works is it basically grabs it it basically gets a pattern right over here and then we just match that pattern against all of the keys in our database and for every key that matches that pattern we return it to the to the client so basically if I have something like let's say if I have something like name and an age in my database I can run something like keys name to get the name key back I can also run something like n a and then a wildcard character which basically means any number of characters after this right and it will still return the name field right the name key so basically what this does is it uses glob style pattern matching right and it uses that to return the number of keys that match that pattern it's going to make a lot more sense once i actually implement this and show it to you all right so let's do that now mind you this is kind of an expensive command to run in that it takes a lot of like server resources depending on how many keys you have so but we don't need to worry about that really because this is like a toy project but in in production settings what you use is instead of using the keys command you use the scan command i might implement that as well in a future part of this video but we'll see so let's create a matches list This will be a string list, right? And then let's loop over every single key in the Database store field over here now There's there's a bunch of different rules for glob style pattern matching and instead of implementing all of those Ourselves which is going to take so much more code than this we can use the file path package So we can do something like file path dot match right and then we just pass in the pattern in here and then just pass in the key name in here this is going to return to fields the matched field and the error field let's let's handle the error first let's lock the error let's say error matching keys let's print the pattern over here and let's also print the key that we try to match over here so we're printing all of the relevant data alongside the error and this is just going to like make it a lot easier for us to debug this whole thing if we need to and let's also print the error as well I guess I believe you do that with the %v but if not you can always fix that later and return after that after printing the error or not return actually but just continue to the next key in the in the database and if if this pattern matches we should append it to the matches list over here append the key over here and I forgot to add the range keyword up here in the loop. Please don't do that. Make sure to add the range keyword So this works Now that all we need to do to grab the matches Finally let create our reply this needs to be a value of type array because that what the redis client expects as the return value in this keys command right so let's create that and then let's loop over all of the matches right and let's append the match to the reply dot array field by doing append reply dot array and then over here let's append a value of type bulk and the bulk will be set to the key name all right which will be this m value in this matches list so basically what we're doing is we're returning an array of key names, right? And then we just return that over here as a reference and remove this colon over here. We're not declaring a new variable. We're just reassigning to an existing variable. So make sure to do that as well. And that's basically all we need to do. Now let's add this up in the handlers list over here, right? Always make sure to do that. there we go save everything and let's try running this now so let's run the server first of all then let's run the client right and let me just try setting a bunch of new keys so let's say key one equals x set key two equals y and set key 3 equals z right so all of these um keys have been set now let's clear everything and try to run the keys command and say something like key and then um a wild card right which basically means any number of characters after this and it's going to return all three of these keys right i can also do something like um if the h key exists right and it does i can do something like a g and then a question mark which basically means only one character after this so any given character after this and if it matches the keys in the database and it does it's going to return the key in a list over here right so basically in this blob style pattern matching you can use symbols like question mark which basically means one character you can use wild card which basically means any number of characters after this or you can just type the complete name of the key and that works as well right you can even do something like something more complicated like something like i don't know n wild card and then e which basically means the n character and then the wild card which basically means any number of characters after this and then finally at the end an e character and this is still going to match the name key in our database so you can see there's a There's a bunch of different rules in this glob style pattern matching, right? This is just the kind of a summary that I'm giving you over here. But there are a couple more tricks you can use to write a bunch more complex patterns to match, right? And instead of implementing all of them by ourselves, we just use the filepad.match function to do it for us, right? and here in the documentation they're showing you a bunch more like characters you can use you can use the wild card you can use the question mark you can use these other symbols as well you can go down this rabbit hole and study this more if you want to but this isn't really relevant to the project so i'm going to skip that but yeah basically that's the keys command yeah there we go so that's the end of this part of the series uh let me just go over everything again we in this part we finished implementing the concurrency part of this server right we fixed all of the server crashes we set up the server to accept and interact with multiple clients at the same time over here right by using a go routine there we go by using a go routine and this handle connection function and everything and then in the handlers list over here we created these three new commands We created the keys command, we created the exists command, and we created the delete command, right? So this server is coming along very nicely. This project is coming along very nicely. We're very far into this, right? In the next part, I'm not sure what I'm going to do in the next part yet, but I am planning to implement the save and the bg save commands, right? So let me just give you a quick overview. of what these commands do remember that the when i was talking about the data persistence methods right um back in a previous video i mentioned that the there are two ways to save an rdb snapshot one is to actually have this save keyword in the configuration file and have a bunch of like conditions after that and this is basically the automatic way to save an rdb snapshot but the save command over here this is a manual way to um to save an rdb snapshot so i could run something like i don't know the save command over here and this would um create an rdb snapshot manually right then and there right and save is also a very expensive operation to run because you're basically looping over all of the like the entire um database store right and if you have something like thousands or millions of keys in your database which is pretty common i guess for really large applications really like large enterprise servers right then it can hang your database for a very long time right and that is why we can also use bg save which basically just stands for background save and what this does is it doesn't block the actual database and it just saves the rdb snapshot in the background right so i'm going to try to implement both of these in the next part of the video. So stay tuned for that. Thank you for watching this part. I hope you learned something. I hope you're building this project alongside me and learning all sorts of new things just as I am. Yeah, thank you for watching. Like, subscribe, share the video, comment if you have any questions or something or if you just want to say hi or whatever. And I will see you in the next video. Thank you for watching this one. Stay tuned for the next part. I'll see you in the next one. Bye-bye.