Auto-generated transcript Okay guys, welcome back to this Go Redis Clone series. We're going to continue building this Redis database clone in Go. Let me just recap the previous part real quick. So, in part 2 of this series, we... Let me expand this. There we go. So in part 2 of this series, we built the command handlers like this handle function, the get command, the set command And we also built this writer struct, right? And what this does is it basically just converts the value struct into a REST string that we can then send to the Redis client. And then it can understand it and parse it and so on and so forth, right? So that's what we built in part two of this series. Now, in part three, which is this part, today what we're going to build is we're going to add concurrency support to this whole program. program so that one server this one server can support multiple clients at the same time right so we're going to add concurrency support to this whole redis server that we've built and secondly we're going to create a configuration file that we're then going to use in data persistence all right so redis obviously has a configuration file that that you can fill out and you can add some options there to like store data in various ways. We're going to create a minimal config file like that and we're going to create a parser for that config file and then we're going to set up data persistence. So those are the two things we're going to build today. Concurrency support and config file and parsing. So let's get started. So let me just run the current server that we have and there we go it's running awesome so the first thing we need to do is create a database right so let's let's get started with that if I go down over here you can see we have a little map over here that we're using as a database but in order to add concurrency support we're going to need it to be a bit more advanced so let's see I'm going to create a database struct all right and this is going to have two fields first the store which is where we store all of our data and this is just going to be a map of strings right just like this database up over here and secondly we're going to have a field called MU this is the mutex we're going to use a read write mutex so you can lock both reading and writing separately all right so this is the database struct that we're going to use. Now let's create a function new database that's going to actually create this and give it back to us. Alright it's going to return a pointer to the database and there we go. So we're returning a pointer to the database. For the store we're just going to initialize with a empty map and for the mutex we're just going to create an empty read write mutex. There we go. Alright now instead of this var DB I'm going to remove that and we're going to have another variable DB which is going to be this new database. Alright now this is going to break our set and get commands because they don't expect this struct over there right so we should go and change that first of all change this to DB dot store and down here as well DB dot store so we're using the correct value and secondly we need to now use the actual mutex to lock and unlock access to this whole database while it's being used alright so the way to do this is very simple you just go DB dot mu dot read lock because we're using get over here so we're going to only only lock reading and then as soon as we've got the value right we can unlock the mute text and let the other clients access this value as well we're going to do the same thing in this set function but instead of just locking reading we're also going to write we're also going to lock writing as well so to do that just do db.mutex.lock, not readlock, lock. And then as soon as you've set the value, you can unlock it so that other clients can then access it as well. Simple as that. That's literally all we need to do to add concurrency support. Now, any number of clients can join and access these commands, and it's not going to cause any race conditions or, you know, any concurrent bugs like that where you're trying to update the same value at the same time or you're trying to access a value that's being overridden at the same time or any of that nonsense right we've prevented all of that by using a simple mutex now one more thing we need to do is up here in the main function we need to actually accept multiple commands multiple connections so if you see right here we're only accepting one connection and just using that right we need to also be able to accept multiple connections over here. Now I'm not gonna bother with that right now because I'm only using one single client right because I'm only building this on my local machine. So I'm not going to bother with that right now but we are going to build this in the future. So for now we have concurrency support. That's amazing. Moving on to the next part we need to now figure out the config file and parsing it and so on and so forth. All right so first of all let's just create a very minimal Redis configuration file. I'll call this Redis.conf and there's a couple things we need to add over here. Alright, first of all the directory where we're going to store all of our data So we going to build data persistence in the next video right We need to have a directory where we store the data files So let add a dir command over here and I'll just say store it in the current directory in a data folder. Next we need to add configuration for the actual data persistence methods. So Redis has two data persistence methods that we can use. There's AOF, append only file and there's also RDB right they're both different in how they work and how they store and retrieve data we're going to build and configure both of them right so first of all let me just configure append only right to enable this you need to use append append only yes and this will enable append only data persistence in your database all right then you can use append file name to set the file name of this database file. So I'll just call this backup.aof and then you can use append fsync and we can set this to every second. Fsync is basically a system call that's used to take data in any buffer and write it to the actual file or the actual connection or whatever right. I'm going to explain this a lot more once we actually configure this but for now just keep that in mind. Next let's also configure the rdb storage. I'll add a comment here rdb. This is going to use the save keyword. We're going to say if in 900 seconds one key changes then we should save the rdb snapshot. All right we're also going to say if in 300 seconds 10 keys change then we should also save the rdb snapshot this basically means the the save command basically tells you when to save the rdb database right when to create a backup of the entire database and store it in disk the 900 value after that is the amount of seconds that need to pass and the one after that is the amount of keys that need to change so what this is basically saying is if in 900 seconds one key changes in your database then save the database right and down here it's just saying if in 300 seconds which I guess is five minutes 10 keys change then you should save the database as well that's what's going on over here and we're going to configure all of this as well finally let's create a db file name and this is going to be the name of the file that's storing the rdb snapshot I'll just call this backup.rdb and with that we are done writing this config. That's all we need to write for now. Now we can actually start to parse this. So let's create a parser for it. I'm going to create a config.go file or just conc.go package main obviously and then let's start to write a parser for this config file. First let's create a config struct right and let me just keep this config file open on the side so you know what I'm building over here. I'm going to create a struct called config. Now this is going to have an RDB field and this is going to be a bunch of RDB snapshots right so stuff like this. So let's create a struct to represent that as well. I'll call it RDB snapshot. Alright we'll say seconds which will be an int and keys change which will also be an int and this RDB field in this config struct is just going to be a list of RDB snapshots there we go I'll also say RDB F F N which is file name this is going to be a string and this is going to be the DB file name over here right backup .rdb then up here let me also add the directory which is going to be a string and this is going to be the data directory up here right then let's add the append only file stuff over here right so let's say aof enable which will be a boolean this is over here then aof file name which is backup.aof over here and this is going to be a string and then aof fsync mode right this is going to be an fsync mode object this is going to be something that we create over here so let me create a new type we'll call this fsync mode this will be a string and now let's create three new constants to represent this so one of the modes is always right so always sync the file every time we write something the other one is every second which is what we're using right here right now right and this will just sync the file every one second right and finally we have no which which basically means that we'll let the operating system handle the syncing process and not interfere with it at all. All right. So those are the three Fsync modes that we're going to use. Now we're pretty much done with representing all of the fields over here. Now we just need to build a parser. So let me create a new config function as well. This will return a pointer to the config. Return config. There we go. And with that we can now start to write the parser. So let me create a read configuration function over here. This will take a file path which will be the path to the config file. And this will return a config object. Right? A pointer to the config object. So let's first create a config object from this new config method that we just created. And then let's open up the config file using os.open. We'll pass in the file path. This will return a file and an error So let handle the error as well If you can read the file let just say cannot read and then we let use printf over here so i can use this percentage syntax cannot read file name using default config so if we can't read the file name then we're just going to use the default configuration right and make sure to add a new line character over here. Let's pass in the file name that we got and then just return the config right. Now down here let's create a defer to close this file so we don't have any open files at the end of this function. Now we can actually start to read the lines in this file so for that let's create a scanner I'll say buff io.newScanner and the reader will be the file, right? So for s.scan, this is going to return new lines. We'll say l equals s.text. This will return a single line and then we need to parse this line, right? So to do that, I'm going to use a new function. But before I do that, let me just finish writing this function. We'll say if there's an error in this scanner, then we should render that over here. You just log it. Let's say error scanning config file and just print that over here and return. Let's return whatever we have in the config object right now. Now let's also make sure that we create the data directories as well. So this data folder over here that we have in the config file, let's make sure that we create this over here So to do that, let's just say if conf dot directory does not equal to an empty string So we have an actual folder name over here, right? Then we should create all of those folders. So let's say OS dot MK dur all and then we'll just pass in the conf dot directory as as the path and for the permissions I'll say 0 7 5 5. This will basically give us the creator full permissions on this whole directory and for everyone else I believe it's only going to give them read and I guess execute permissions but not write permissions. So we make sure to do that as well and finally we can just return the config object once it's been filled up right. So now we actually need to parse these lines. I'm going to create a function called parse line. This is going to take the actual line which is a string and it will take the config object as well which is a pointer to the actual config object so we can actually update this object as much as we need to. So now let's create this function parse line it's going to have a line which is a string and a config which is going to be a pointer to the config object. It's not going to return anything. Now, before we actually write this line parser, let's take a moment to analyze this config file and see just how the configuration is arranged, right? So, you can see that in every single line, first of all, we have a command or a keyword, whatever you want to call it, right? That tells us what this configuration is going to do. And then we have a bunch of arguments, right? So we can basically just scan each line, separate every single word by a space, right? And then let's just use a switch statement to grab all of these keywords and parse them, right? If this isn't making sense to you, then just follow along and you'll see what I mean. So first of all, what we need to do is let's say, let's create an args variable. and this is going to use the strings.split method to split the line by a space, right? And the actual command is going to be the first argument, right? So something like dir or append only or append file name, save, db file name. These are the actual commands, right? So these are going to be the first item in this args list. We're going to grab that and then we're going to create a switch statement on this command. The first case is going to be save, right? This one over here. Now, if we get the save command over here, we know that the second argument is going to be the seconds and the third argument is going to be the keys changed, right? So, let's grab those. These are going to be in strings, right? These are going to be a type of strings. So, let's also convert them into an integer so we can actually store them right the seconds are going to be in the second argument make sure to handle the errors always let's say invalid seconds and then return then let's also grab the keys changed and do the same thing over here these are going to be the third argument right let's handle the error let's say invalid keys over here and then return if there's an error and if it's all going well no errors then we can actually create the snapshot that we need, pass in the values, right? And then we can just append this to the actual config object, right? And that's all we need to do. So next we can handle the DB file name right And this one is super simple If we get this then we just need to grab the second argument here which is the file name And we can assign that to the where is it the rtbfn field All we need to do is just pass in the second argument. There we go, right? Similarly, in append file name, we can just do the same thing, but to the aoffn field, right? As for the append fsync field, we can just grab this field over here and convert it into an fsync mode object. Since both of these values are strings, it's easy to convert one to the other, right? So pass it in over here and that's all we need to do. Now what do we have left? Okay, we still need to handle the append only keyword and the dir keyword. Okay, so let's do that as well. Let's say dir, on.dir is going to be the second argument over here. Finally, let's handle the append only keyword. say if the second argument equals yes then we can enable this right set it to true otherwise we set it to false all right super simple stuff and that's all we need to do all right now we built the parser completely now we just need to actually use it right so let's go back to our main function over here and let's Let's read the configuration file first thing, right? Before we set up the listener, before we set up the server, let's first read the configuration file. We'll pass in the path to the configuration file, which is just going to be redisk.conf in the current directory, right? Of course, you can store this anywhere you want, obviously. I stopped the server. Let me add a bunch of logs over here as well. just so we know that this is actually working fine, right? We'll say reading config file. All right. Let's run this and see if it works. There we go. Everything works. Awesome. And you can see it also created this data folder for us, right? So you can tell that everything is working fine. This data folder wasn't present here before, right? it created it for us after after reading the entire config file so everything is working perfectly over here now these files are getting a bit too long especially this main file right there's a bunch of different unrelated stuff over here so I want to take just a couple minutes to organize this whole thing into a bunch of different files alright just so everything is a lot more clearer and everything right so let me just grab the entire rider struct the new writer method and the write method right and create a writer dot go file this will again be in the package main and there we go we'll just paste everything over here save changes and that's a bit better right now let's copy all of these handlers right all the way up here and create a handlers dot go file again package main and paste them in here and save everything there we go let's also move this handlers list and this type to this file this handlers dot go file there we go awesome now the database as well let's move that to a I I guess tb.go file, right? And make sure to add the package main up at the top. And I think that's about it. Let's also move this value struct and all of its related types and the value type as well to a value.go file. There we go. Save everything and awesome. Let's Let's run the server again to make sure everything is working fine and we didn't break anything. And perfect, it's running. And let's also run a client over here. I'll say Redis CLI. Awesome, so that's working as well. Let's try to set the name. That works. Let's try getting it. That works as well. Let's try setting an age. That works. it also works awesome so that's the end of part three of this series we built concurrency support by using a mutex a read write mutex we also created a parser for this config file over here and now we actually have a config object that we can use to create data persistence right so depending on what options we set over here we're going to enable or disable or just mess around with data persistence stuff right all of that stuff we're going to do in the next video right in this video we built this config parser and added the concurrency support i'm gonna end the video over here um before i do that though i wanna just say a quick thank you for supporting this series i uh i checked out the channel videos just like yesterday and it seems that this video series It's getting a lot more views than my usual videos. So clearly there's a lot of interest over here, right? So I'm gonna continue making videos on this whole topic on building this redis clone in go because clearly there's a lot of people watching it as well, so thank you for the extra views and everything and Yeah, that's the end of this video Thank you for watching like subscribe all of those wonderful things do all of them drive up the algorithm the next part is going to come out in a day or two so stay tuned for that and yeah thank you for watching bye