Building a Redis clone in Go - Part 16 (Fixing AOF file sync)

Hey, what's going on, guys? This is part 16 of the GoRedis clone series. In the previous part, we implemented eviction policies like allkeys-random. Where is it? Let me just show you. There we go, the allkeys-random eviction policy, and we just did a bunch of other features. We're going to continue building the eviction policies, all of the ones that we've configured over here. But before we do all that, I want to fix a very major issue in our server right now, and that is in the AOF file over here. So let me just show you what the problem is. First of all, let's run the server, and you can see over here that I have three keys in my AOF file over here. We have the name key, the age key, and the job key. All right, now watch what happens over here. If I run KEYS, you'll see that only the name key is present, right? So it didn't even bother to back up and restore the age key and the job key, right? So there is a problem over here. The problem is that the AOF file, when we synchronize it back to the server, when we restore the database from this AOF backup, it only restores the first key in the file, right? It ignores all of the other keys after that. So before we continue implementing all of the eviction policies, I want to just fix this error inside the AOF file, and we can then continue implementing all of the eviction policies in future videos. So let's do this. So the first thing we need to do is go over to the, where is it, value file over here, and you see all of these readArray methods and readBulk method over here. The problem, one of the problems, is that we're creating a new reader every single time we want to read an array, right? Now, in network connections and in clients over here, this doesn't really cause a problem because we only ever send one key or one command at a time, right? In a Redis client, we only send one command at a time, so creating a new buffer over here is not a problem. But in something like the AOF file, where we have multiple different commands all at the same time, creating a new reader for them is a bit of a problem, right? So if I show you the AOF file over here and go down to the synchronize function, you can see that we're reading the array and passing in the AOF file over here, right? And this will create a new buffer reader for that AOF file every time we read a new array. It's going to create a new reader, and it just breaks the sync between the files, right? Because you read one array over here, and then you create a new reader, and then you try to read the next array, and it just becomes a mess of things, right? So instead of creating the reader over here, delete this. Change the argument to R, which will be above I/O dot Reader pointer, and let's pass in our own reader over here. So in the AOF file, let's say R equals bufio.NewReader, pass in the AOF file over here, and pass in the R reader in this V.ReadArray method call, right? Similarly, in the main.go file up here, let's create, in the handleCon function, let's create a reader called bufio.NewReader and pass in the connection as the reader, and pass the R variable in the readArray function over here. So that's the first thing we need to do. Secondly, we want to fix some issues in our readBulk method over here. We want to basically make this able to return values as well as errors. So instead of having to, you know, basically log all of these errors and do nothing with them, let's return them as well. There we go. And down here, just return nil if there is no error, and handle this over here as such. There we go. And over here, let's just print the error because there's not much else we're going to do with it, and add a continue over here. So if we can't scan one of the bulk strings, we should just continue over to the next one, or even not even continue, but just break, because if you can't scan one, then there's no point in scanning the next one because the entire command is going to be invalid anyway. So let's try to run the server now again, and this time it's going to work. You can see over here the memory logs, right? It increases three times, from 98 to 191 to 292. So you can already tell just from this that we're loading three keys from the AOF backup. If I run the KEYS command, you can see that all three of these keys were now backed up. Awesome. So we fixed all of the issues in our AOF sync function over there. Now I am going to keep this video short so it's easy for people to watch. So I'm going to end the video over here, and in the next video after this, we're going to continue building eviction policies. I believe in the next video we're going to implement the LRU and the LFU eviction policies for all keys. I'll try to do both of these in the next video, and then after that, we can implement all of the volatile eviction policies as well. So stay tuned for that. Thank you for watching this one. Like, comment, subscribe, and I'll see you in the next one.