Building a Redis clone in Go - Part 12 (Rewriting AOF files)

Welcome back, guys. We're going to continue building this Go Redis clone. In the previous video, we built the EXPIRE and the TTL commands over here, and these will just allow us to expire keys after some time, right? Not gonna go too deep into them again. In this video, we're going to build the BGREWRITEAOF command. Let me just add it over here. BGREWRITEAOF command. There we go. And what this is going to do is it's going to rewrite our AOF file over here and just make it more compact. So let me first show you the issue over here that we have, right? Let me delete everything that we have right now. Let's comment this out, and there we go. Let's run the server. I just want to show you what the problem is right now before I actually solve it, right? And let me just, before I actually run this server, let me disable the password protection because I just hate having to enter the password every time I run the client. And let's set the name key to, let's say, Hasan. All right. So you can see that it backs it up in the AOF file, right? Now, what happens if I change it to Michael? And it backs it up again, right? And if I change it again to, I don't know, Elon. How's that? You should be able to see a problem over here, right? Every time we make a change, it gets backed up into the AOF file, which is great. But at the same time, when we actually synchronize this to the database, right, it's going to first set the name key to Hassan, then to Michael, and then finally to Elon. The thing is that the final value of the name key is the string Elon, right? And all of these other SET commands before it are completely unnecessary, right? So the server is literally just executing these SET commands for no reason because the final value is just going to be Elon over here, right? And that is the problem with this AOF function, with this AOF data persistence method over here, right? Because it just records all of these SET commands, and they can become outdated as we override different keys with new values, right? So if I run the server again, the value of the name key should be Elon, right? Not Hasan, not Michael, but Elon. Okay, which means we don't really need all of these other SET commands up here. That is what the REWRITEAOF command is going to fix, right? So the BGREWRITEAOF command is going to run in the background. That's why BG, right? And it's going to rewrite the AOF file to remove all of these old and outdated SET commands and just have the latest version of them, right? And it's just going to make our whole AOF file much, much smaller, much more compact, and much easier to read and use. And it's going to improve the performance of the server because it doesn't have to execute all of these extra unnecessary commands to backup and restore the database. Alright, that's what we're going to build, and now that you're all up to speed, let's get started. So let's uncomment this, and down here, create the command handler for this. Let's say BGREWRITEAOF. Just like all the other command handlers, this is going take the client and the value and the state. There we go. And it will return a pointer to a value. Now, since this is supposed to run in the background, we should start a goroutine over here and run this over here. And let's first of all lock the database. Unlock it down here and create a copy of the database. Call it CP again. It's just CP. And this is going to, what is it going to take? Let me just... Yeah, it's going to take the maps.Copy, actually, not, not, not that. It's going to take the make command, the make function. I don't know why I call functions commands and commands functions. I'm just fucked in the head. But we're going to make a map with keys as strings and values as key pointers. There we go. And this will have a length of, what is a DB.Store? Then we're going to use maps.Copy to copy the entire database into this CP variable. And once this is done, we can unlock the database and carry on. And by the way, we can change this lock to a read lock and down here as well to a read unlock because we're only reading the database values, right? We're not modifying them in any way. So that's done. Now we can actually just run the state.AOF.Rewrite function, which doesn't exist yet. We need to write it over here. But this function is basically going to rewrite the AOF file, open it up, and just rewrite all of the SET commands for us, right? So let's call that over here. It will take the copy that we create over here, and then we can just return a value up here of type string, and we can say, "Background AOF rewriting started." Awesome. So now let's actually create this rewrite function, and it'll take a map of string and a key pointer. There we go. So first of all, what we want to do is we want to reroute all of the existing AOF records to the, to a memory buffer, basically. So this AOF file is going to be busy now, right? Because we're going to rewrite it. But what happens if one of the clients that's running sends a SET command, right? Like SET name Hasan or something, right? That new command still needs to be recorded in the AOF file, right? And in order to do that, since we're going to be using the file because we're rewriting it, right, the file is going to be busy. We want to reroute all of the future, basically all the future SET commands to a memory buffer that we can then write to the AOF file later on when we're done rewriting it. So let me just add a comment over here as well. Future SET commands to buffer. And instead of following the SET commands, let's just call this AOF records. Alright, let's create a buffer. There we go. And the AOF writer will just be a new writer to the buffer over here. Awesome. Now let's clear the file contents, right? Because we want to delete everything in this file right now and rewrite it. So clearing the file content is going to be a two-step process. First of all, we want to run the AOF.F.Truncate function and just add zero over here. What this is going to do is it's going to change the file size, right, to zero, and it's going to return an error. So let's grab that as well and make sure to handle it over here. If there's an error, let's just log it. Let's say AOF rewrite truncate error, and then log the error over here and return. So that's the first thing. This is going to shrink the AOF file to zero bytes, right? Delete all of the data inside the AOF file. Then the next thing we want to do is we want to seek the AOF file to zero bytes. All right, so basically we've talked about byte cursors in a previous video as well. Basically, we want to make sure that the byte cursor, the point where we start reading or writing to a file, right, is at the beginning of the file. So we do that with this seek function. Make sure to handle the error as well. Let's say AOF rewrite seek error, and then just log the error over here and return. Why is this giving me an error? Okay, because it returns two values, so let's just ignore the first value by using an underscore. There we go. And now we've cleared the file contents, and the AOF file is going to be completely empty. Now we can start the rewriting process. So what we can do now is create a file writer and create a new writer over here and pass in the AOF file as such. This new writer will just create a writer struct, which is what we created in one of the earlier parts of this series. And what this will basically allow us to do is we can serialize all of the values that we pass in into RESP strings and write them to the file directly. All right. So with that done, now we can loop over the entire database copy that we get over here. Right. Now, pay attention. How exactly does Redis send commands to the server? It sends them as an array of bulk strings. Right. So let's create that. All we need to create the command, right, which will be a value of type bulk because all Redis commands are a set of bulk strings, right? An array of bulk strings. So the command will be, where is it? Bulk set, right? Because all of these are going to be set commands, right? Then we want to create the key, which will be a value of type bulk again, and this will be the k variable over here, right? Then the value, again type bulk, and this will be v.v. So this v is a key struct over here, right? And inside it, we have another v variable, this v field, which will be the actual string value over here, right? Right here. So now that we have all of these, now we can create an array, which will be a value of type array, and the array will just be an array of values. We'll pass in the command, the key, and the value. There we go. And now we can actually just write this over here by saying fwriter.write and pass in the array as a pointer. This is going to write the command by converting it into a RESP protocol message, right? And then write that to the file. Now, once all of the keys and values in the database copy have been written, then we're just going to flush the output just once, right? And this is going to make sure that the bytes that we write actually end up on the file, on disk. And now the entire rewriting process is complete. I'm actually gonna add a comment over here as well: rewrite all SET commands, or actually just write all SET commands to file. All right, awesome. Now we're done with the entire rewriting process. Now we should reroute all of the future AOF records from the buffer back to the AOF file over here. So let's say, reroute future AOF records back to file. There we go. And let's just say AOF.writer equals new writer and pass in the AOF file over here. And with that, we are done with this rewrite function completely. Now, we're basically done with all of this, but there is one more issue to solve, and that issue is that we have actually not been using the correct code to parse the RESP strings over here. So the read array method and the read bulk method, all of these that we created in, I guess, the first or second part of this series, have been wrong. All right, I'm sorry, I just messed up over here, and I did not use the... I just didn't build these in the correct way. So let me first show you what the problem is, and then I'll explain what it is, and then we'll solve it. All right, step by step. Ready CLI, open up a client. Let me just show you what happens if I run BGREWRITEAOF. It's going to return this STRCONF error over here, right? This parsing error in the server, you can see, right? The problem over here, let me just explain, is this BGREWRITEAOF command is 12 characters, right? You can see: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12. It's 12 characters. Why is that a problem? Because our read array method and our read bulk method can only support strings of less than 10 characters. Why? Because we're directly accessing the first byte in the buffer over here to grab the length of the array and the length of the bulk strings. Why is that a problem? Because we're only accessing one byte over here, right? And one byte can only handle one character, right? So 1, 2, 9, right? It can't handle two values, two bytes, like 10 or 11 and so on and so forth, right? So that's a problem, and we need to fix that. Otherwise, we'll just never be able to use strings or values of, you know, more than nine characters, right? Ten characters and beyond. So let's fix that. So first of all, let's just create a helper method over here. Call it readLine. And this is going to take a reader, which will be a bufio.Reader as such. And it will return a string, which will be the line that it reads, and an error. Then let's say line equals r.readString and pass in the delimiter, which will be '\n'. This will also return an error. So let's make sure to handle that as well. And by the way, instead of using double quotes over here, let's use single quotes, so it doesn't give us an error over here. Awesome. Now, if there is an error, let's return an empty string and the error. Otherwise, let's return the line and nil. And instead of returning the line directly, what we're going to do is we're going to use the strings.TrimSuffix function. Let me import this package over here. TrimSuffix function, and we're going to trim the '\r\n', the newline characters at the end. And this way, we won't have to remove these newline characters on our own, right? This is just going to return us the line without these newline characters. So it's going to be very, very useful for us. Now, first of all, in the read array method, let's create a new bufio.Reader by saying bufio.NewReader, passing the reader that we get over here. So basically, we're gonna get a reader in the read array method, and we're going to convert it into a buffered reader using this method. Alright then, read the first line by calling in the read line function and passing in this reader. Make sure to handle the error as well. If there is an error, let's just return it. And let's fix this as well. So go up here to the read line method, and instead of accepting a `bufio.Reader`, let's accept a pointer to `bufio.Reader`. Awesome. Now let's add a bunch of error handling over here. Let's say if the first character in the line is not the star symbol, which is the symbol for arrays, use single quotes over here, guys. I don't know why I keep forgetting that. If this is the case, then we can just return a new error over here and say, "Expected array," and got something else. Right now, we don't need to use this buffer anymore. We can remove all of that, and instead of using this in the `strconv.Atoi` call over here, right, instead of only accepting the first byte, we can just say `line` and then just accept everything after the first character, right? So if you remember what REST strings look like, right, it's going to be first of all the array symbol, which is star, then the length of the array, let's say it's three, and then backslash r backslash n, right? What we're doing now is we're reading the first line using this read line method. We're trimming the backslash r backslash n, so we can remove them over here, right? All that's left now is the star symbol and the number, right, the length of the array. So we ignore the first character, which will be the star symbol, and then everything after that is just going to be the length of the string, right? So now we can just use this function over here, and even if the length of the array is 10 elements or 20 elements or 500 elements, it's not going to matter. It's going to read it properly this time. Everything else is perfectly fine over here, but we need to make the same changes in this read bulk method as well, so it can accept strings of greater length than 10 characters, right? Now, this read bulk method is only being called in this `v.readBulk` over here in this read array method, right? This should accept a buffered reader as such, and let's change this to a `bufio.Reader` as well over here. Now, we don't need to read these directly. We can just say `line, error` and just use the read line function that we wrote. Pass in the buffer. Make sure to handle the error as well. And let me just log the error over here because we don't actually return anything over here. Error in read bulk, and just log it over here and return, and you can return an empty value. Doesn't really matter. There we go. And then we just need to, instead of passing in this buffer and the first byte, we can just say `line`, ignore the first character, which will be the type of the buffer, type of the bulk string, which will be the dollar sign, and just grab everything after that, which will be the length of the buffer string. There we go. Now, instead of using `reader.Read` over here, let me just rename this buffer to `buff`, just like that. What we can do is we can use the `io.ReadFull` method, which will basically just read and keep reading until we fill the buffer, right? So what we can do is we can say `_error`, `io.ReadFull`. We can pass in the reader, which is going to be `r`, and then the buffer, which is going to be `buff` over here. Make sure there's no error. If there is any, we can print it and return value over here. By the way, this is not like best practice, to not return the errors like this. I'm just doing it because I know this is going to work, right? And this is kind of like a practice project. But in a production scenario, what you should do is you should return the error value so that the caller, in this case, the read array method, right, can use it to basically just do whatever it wants, right? Whether that's rerunning the server, retrying the function or something, or just crashing or whatever, right? But we should return an error value over here so that the caller can use it in whatever way seems best, right? I'm not going to do that over here because this is just a practice project, but I'm just saying it's best practice to return the error. Anyway, now we're done with all of this. Let's run the server again and see if everything went well. Let's run the BG rewrite AOF command again, and it says, "Background AOF rewriting started." Let's go over here, and you can see it actually did rewrite everything for us, right? So all of the SET commands for the name key are now overwritten. Let's actually try setting the name value again. Let's say name one. Then let's say name two and name three. All right, so the latest value should be name three, right? Let's say BG rewrite AOF again, and it removed all of them and only gave me name three over here, right? Which is the latest value. So that works as well. The BG rewrite AOF command works, and we can now get a much more compact, much more shorter, and cleaner AOF file that doesn't take up as much space, that doesn't take too much time to synchronize with the server, and it's just so much easier to maintain, right? So yeah, that's it. We've built this. That's the end of this part of the series. Thank you for watching so far. Make sure you like, share, and subscribe, and comment, and all of those wonderful things that drive up the YouTube algorithm for me. And thank you for following along with this series so far. It's been like, I think this is part 12 of the series, so 12 videos on just this one project so far. So if you're still watching this, thank you. Just thank you so much for supporting the series and watching it for so long, right? Hopefully you're learning new things just as I am, and hopefully you're getting some sort of value from this video series, right? I'm going to continue this for as long as I can, but yeah, so far so good. Thank you for watching. Like, share, subscribe, and I will see you in the next video, in which we are going to build database transactions. That's going to be a very kind of complicated topic, but also a very interesting and fun topic to just work on and build, right? So yeah, stay tuned for that, and I will see you there.