Building a Redis clone in Go - Part 3 (Concurrent Server, Config Parser)

Okay, guys, welcome back to this Go Redis Clone Series. We're going to continue building this Redis Database Clone in Go. Let me just recap the previous part real quick. So, in part 2 of this series, we... Let me expand this. There we go. So, in part 2 of this series, we built the command handlers like this handle function, the get command, the set command, and we also built this writer struct, right? And what this does is it basically just converts the value struct into a RESP, a RESP string that we can then send to the Redis client, and then it can understand it and parse it and so on and so forth, right? So that's what we built in part two of this series. Now, in part three, which is this part today, what we're going to build is we're going to add concurrency support to this whole program so that one server, this one server, can support multiple clients at the same time, right? So we're going to add concurrency support to this whole Redis server that we've built. And secondly, we're going to create a configuration file that we're then going to use in data persistence, all right? So Redis obviously has a configuration file that you can fill out, and you can add some options there to, like, store data in various ways. We're going to create a minimal config file like that, and we're going to create a parser for that config file, and then we're going to set up data persistence. So those are the two things we're going to build today. Concurrency support and config file and parsing. So let's get started. So let me just run the current server that we have, and there we go, it's running. Awesome. So the first thing we need to do is create a database, right? So let's get started with that. If I go down over here, you can see we have a little map over here that we're using as a database, but in order to add concurrency support, we're going to need it to be a bit more advanced. So let's see. I'm going to create a database struct, all right? And this is going to have two fields. First, the store, which is where we store all of our data, and this is just going to be a map of strings, right? Just like this database up over here. And secondly, we're going to have a field called MU. This is the mutex. We're going to use a read-write mutex so you can lock both reading and writing separately. All right, so this is the database struct that we're going to use. Now let's create a function, new database, that's going to actually create this and give it back to us, all right? It's going to return a pointer to the database, and there we go. So we're returning a pointer to the database. For the store, we're just going to initialize with an empty map, and for the mutex, we're just going to create an empty read-write mutex. There we go. All right, now instead of this var DB, I'm going to remove that, and we're going to have another variable DB, which is going to be this new database. All right, now this is going to break our set and get commands because they don't expect this struct over there, right? So we should go and change that. First of all, change this to db.store, and down here as well, db.store, so we're using the correct value. And secondly, we need to now use the actual mutex to lock and unlock access to this whole database while it's being used, all right? So the way to do this is very simple. You just go db.mew.read lock because we're using get over here, so we're going to only only lock reading. And then, as soon as we've got the value, right, we can unlock the mutex and let the other clients access this value as well. We're going to do the same thing in this set function, but instead of just locking reading, we're also going to write. We're also going to lock writing as well. So to do that, just do db.mutex.lock, not readlock, lock, and then as soon as you've set the value, you can unlock it so that other clients can then access it as well. Simple as that. That's literally all we need to do to add concurrency support. Now any number of clients can join and access these commands, and it's not going to cause any race conditions or, you know, any concurrent bugs like that, where you're trying to update the same value at the same time, or you're trying to access a value that's being overridden at the same time, or any of that nonsense, right? We've prevented all of that by using a simple mutex. Now, one more thing we need to do is up here in the main function, we need to actually accept multiple commands, multiple connections. So if you see right here, we're only accepting one connection and just using that, right? We need to also be able to accept multiple connections over here. Now, I'm not gonna bother with that right now because I'm only using one single client, right? Because I'm only building this on my local machine, so I'm not going to bother with that right now. But we are going to build this in the future, so for now, we have concurrency support. That's amazing. Moving on to the next part, we need to now figure out the config file and parsing it and so on and so forth, all right? So first of all, let's just create a very minimal Redis configuration file. I'll call this redis.conf. And there's a couple things we need to add over here, all right? First of all, the directory where we're going to store all of our data. So we're going to build data persistence in the next video, right? We need to have a directory where we store the data files. So let's add a dir command over here, and I just say store it in the current directory in a data folder. Next, we need to add configuration for the actual data persistence methods. So Redis has two data persistence methods that we can use. There's AOF, Append Only File, and there's also RDB, right? They're both different in how they work and how they store and retrieve data. We're going to build and configure both of them, right? So first of all, let me just configure append only, right? To enable this, you need to use appendonly yes, and this will enable append-only data persistence in your database, all right? Then you can use appendfilename to set the file name of this database file. So I'll just call this backup.aof, and then you can use appendfsync, and we can set this to every second. Fsync is basically a system call that's used to take data in any buffer and write it to the actual file or the actual connection or whatever, right? I'm going to explain this a lot more once we actually configure this, but for now, just keep that in mind. Next, let's also configure the RDB storage. I'll add a comment here, RDB. This is going to use the save keyword. We're going to say if in 900 seconds one key changes, then we should save the RDB snapshot, all right? We're also going to say if in 300 seconds 10 keys change, then we should also save the RDB snapshot. This basically means the save command basically tells you when to save the RDB database, right? When to create a backup of the entire database and store it in disk. The 900 value after that is the amount of seconds that need to pass. And the one after that is the amount of keys that need to change. So what this is basically saying is if in 900 seconds one key changes in your database, then save the database, right? And down here, it's just saying if in 300 seconds, which I guess is five minutes, 10 keys change, then you should save the database as well. That's what's going on over here. And we're going to configure all of this as well. Finally, let's create a DB file name. And this is going to be the name of the file that's storing the RDB snapshot. I'll just call this backup.rdb. And with that, we are done writing this config. That's all we need to write for now. Now we can actually start to parse this. So let's create a parser for it. I'm going to create a config.go file, or just conc.go. Package main, obviously. And then let's start to write a parser for this config file. First, let's create a config struct, right? And let me just keep this config file open on the side so you know what I'm building over here. I'm going to create a struct called config. Now, this is going to have an rdb field, and this is going to be a bunch of RDB snapshots, right? So stuff like this. So let's create a struct to represent that as well. I'll call it rdb snapshot. Alright, we'll say seconds, which will be an int, and keys change, which will also be an int. And this rdb field in this config struct is just going to be a list of rdb snapshots. There we go. I'll also say rdb fn, which is file name. This is going to be a string, and this is going to be the DB file name over here, right? backup.rdb. Then up here, let me also add the directory, which is going to be a string, and this is going to be the data directory up here, right? Then let's add the append-only file stuff over here, right? So let's say aof enable, which will be a boolean. This is over here. Then aof file name, which is backup.aof over here, and this is going to be a string. And then aof fsync mode, right? This is going to be an fsync mode object. This is going to be something that we create over here. So let me create a new type. We'll call this fsync mode. This will be a string, and now let's create three new constants to represent this. So one of the modes is always, right? So always sync the file every time we write something. The other one is every second, which is what we're using right here right now, right? And this will just sync the file every one second, right? And finally, we have no, which basically means that we'll let the operating system handle the syncing process and not interfere with it at all. Alright, so those are the three fsync modes that we're going to use. Now, we're pretty much done with representing all the fields over here. Now, we just need to build a parser. So, let me create a new config function as well. This will return a pointer to the config. Return config. There we go. And with that, we can now start to write the parser. So, let me create a read configuration function over here. This will take a file path, which will be the path to the config file, and this will return a config object, right? A pointer to the config object. So let's first create a config object from this new config method that we just created, and then let's open up the config file using os. We pass in the file path. This will return a file and an error, so let handle the error as well. If you can read the file, let just say cannot read, and then we let use printf over here so I can use this percentage syntax. Cannot read file name using default config. So if we can't read the file name, then we're just going to use the default configuration, right? And make sure to add a new line character over here. Let's pass in the file name that we got, and then just return the config. Right now, down here, let's create a defer to close this file so we don't have any open files at the end of this function. Now we can actually start to read the lines in this file. So for that, let's create a scanner. I'll say bufio dot new scanner, and the reader will be the file, alright? So for s.scan, this is going to return new lines. We'll say l equals s.text. This will return a single line, and then we need to parse this line, right? So to do that, I'm going to use a new function, but before I do that, let me just finish writing this function. We'll say if if there's an error in this scanner, then we should render that over here. You just log it. Let's say error scanning config file, and just print that over here, and return. Let's return whatever we have in the config in the config object right now. Now let's also make sure that we create the data directories as well. So this data folder over here that we have in the config file, let's make sure that we create this over here. So to do that, let's just say if conf.directory does not equal to an empty string, so we have an actual folder name over here, right? Then we should create all of those folders. So let's say os.mkdir all, and then we'll just pass in the conf.directory as the path, and for the permissions, I'll say 0755. This will basically give us the creator full permissions on this whole directory, and for everyone else, I believe it's only going to give them read and I guess execute permissions, but not write permissions. So we make sure to do that as well. And finally, we can just return the config object once it's been filled up, right? So now we actually need to parse these lines. I'm going to create a function called parse line. This is going to take the actual line, which is a string, and it will take the config object as well, which is a pointer to the actual config object, so we can actually update this object as much as we need to. So now let's create this function, parse line. It's gonna have a line, which is a string, and a config, which is going to be a pointer to the config object. It's not going to return anything. Now, before we actually write this line parser, let's take a moment to analyze this config file and see just how the configuration is arranged, right? So you can see that in every single line. First of all, we have a command or a keyword, whatever you want to call it, right? That tells us what this configuration is going to do. And then we have a bunch of arguments, right? So we can basically just scan each line, separate every single word by a space, right? And then let's just use a switch statement to grab all of these keywords and parse them, right? If this isn't making sense to you, then just follow along and you'll see what I mean. So first of all, what we need to do is, let's say, let's create an args variable, and this is going to use this string's dot split method to split the line by a space, right? And the actual command, it's going to be the first argument, right? So something like dir or append only or append file name, save DB file name. These are the actual commands, right? So these are going to be the first item in this args list. We're going to grab that, and then we're going to create a switch statement on this command. The first case is going to be save, right? This one over here. Now, if we get the save command over here, we know that the second argument is going to be the seconds, and the third argument is going to be the keys changed, right? So let's grab those. These are going to be in strings, right? These are going to be a type of strings, so let's also convert them into an integer so we can actually store them, right? The seconds are going to be in the second argument. Make sure to handle the errors always. Let's say, "Invalid seconds," and then return. Then let's also grab the keys changed and do the same thing over here. These are going to be the third argument, right? Let's handle the error. Let's say, "Invalid keys" over here, and then return if there's an error. And if it's all going well, no errors, then we can actually create the snapshot that we need. Pass in the values, right? And then we can just append this to the actual Config object, right? And that's all we need to do. So next, we can handle the DB file name, right? And this one is super simple. If we get this, then we just need to grab the second argument here, which is the file name, and we can assign that to the, where is it, the rdbfn field. All we need to do is just pass in the second argument. There we go, right? Similarly, in append file name, we can just do the same thing, but to the aoffn field, right? As for the append fsync field, we can just grab this field over here and convert it into an fsync mode object. Since both of these values are strings, it's easy to convert one to the other, right? So pass it in over here, and that's all we need to do. Now, what do we have left? Okay, we still need to handle the append only keyword and the dir keyword. Okay, so let's do that as well. Let's say dir. On.dir is going to be the second argument over here. Finally, let's handle the append only keyword. Say if the second argument equals yes, then we can enable this, right? Set it to true. Otherwise, we set it to false. All right, super simple stuff, and that's all we need to do. All right, now we built the parser completely. Now we just need to actually use it, right? So let's go back to our main function over here, and let's read the configuration file. First thing, right? Before we set up the listener, before we set up the server, let's first read the configuration file. We'll pass in the path to the configuration file, which is just going to be redis.conf in the current directory, right? Of course, you can store this anywhere you want, obviously. I stopped the server. Let me add a bunch of logs over here as well, just so we know that this is actually working fine, right? We'll say, "Reading config file." All right, let's run this and see if it works. There we go. Everything works awesome, and you can see it also created this data folder for us, right? So you can tell that everything is working fine. This data folder wasn't present here before, right? It created it for us after reading the entire config file, so everything is working perfectly over here. Now, these files are getting a bit too long, especially this main file, right? There's a bunch of different unrelated stuff over here. So I want to take just a couple minutes to organize this whole thing into a bunch of different files, all right? Just so everything is a lot more clearer and everything, right? So let me just grab the entire writer struct, the new writer method, and the write method, right? And create a writer.go file. This will again be in the package main, and there we go. We'll just paste everything over here, save changes, and that's a bit better, right? Now let's copy all of these handlers, right? All the way up here, and create a handlers.go file. Again, package main, and paste them in here, and save everything. There we go. Let's also move this handlers list and this type to this handlers.go file. There we go, awesome. Now the database as well. Let's move that to a, I guess, db.go file, right? And make sure to add the package main up at the top, and I think that's about it. Let's also move this value struct and all of its related types, and the value type as well, to a value.go file. There we go. Save everything, and awesome. Let's run the server again to make sure everything is working fine and, like, and we didn't break anything. And perfect, it's running. And let's also run a client over here. I'll say, "Ready CLI." Awesome, so that's working as well. Let's try to set the name. That works. Let's try getting it. That works as well. Let's try setting an age. That works. Getting it also works. Awesome. So that's the end of part three of this series. We built concurrency support by using a mutex, a read-write mutex. We also created a parser for this config file over here. And now we actually have a config object that we can use to create data persistence, right? So depending on what options we set over here, we're going to enable or disable or just mess around with data persistence stuff, right? All of that stuff we're going to do in the next video. Right, in this video, we built this config parser and added the concurrency support. I'm gonna end the video over here. Before I do that, though, I wanna just say a quick thank you for supporting this series. I checked out the channel videos just like yesterday, and it seems that this video series is getting a lot more views than my usual videos, so clearly there's a lot of interest over here, right? So I'm gonna continue making videos on this whole topic, on building this Redis clone in Go, because clearly there's a lot of people watching it as well. So thank you for the extra views and everything. And yeah, that's the end of this video. Thank you for watching. Like, subscribe, all of those wonderful things. Do all of them. Drive up the algorithm. The next part is going to come out in a day or two, so stay tuned for that. And yeah, thank you for watching. Bye-bye.