Building a Redis clone in Go - Part 16 (Fixing AOF file sync)

Hey, what's going on guys? This is part 16 of the GoRedis clone series. In the previous part, we implemented eviction policies like all keys random. Where is it? Let me just show you. There we go. The all keys random eviction policy and we just did a bunch of other features. We're going to continue building the eviction policies, all of the ones that we've configured over here. But before we do all that I want to fix a very Major issue in our server right now and that is in the AOF file over here So let me just show you what the problem is. First of all Let's run the server and you can see over here that I have three keys in my AOF file over here We have the name key the age key and the job key. All right now watch what happens over here if I run keys you'll see that only the name key is present right so I didn't even bother to backup and restore the age key and the job key right so there is a problem over here the problem is that the AOF file when we synchronize it back to the server when we restore the database from this AOF backup it only restores the first key in the in the file right it ignores all of the other keys after that so before we continue implementing all of the eviction policies I want to just fix this error inside the AOF file and we can then continue implementing all of the eviction policies in future videos so let do this so the first thing we need to do is go over to the where is it value file over here and you see all of these read array methods and read bulk method over here the problem one of the problems is that we're creating a new reader every single time we want to read an array right now in network connections and in clients over here this doesn't really cause a problem because we only ever send one key or one command at a time right in a redis client we only send one command at a time so creating a new buffer over here is not a problem but in something like the aof file where we have multiple different commands all at the same time creating a new reader for them is a bit of a problem right so if i show you the aof file over here and go down to the synchronize function you can see that we're reading the array and passing in the aof file over here right and this will create a new buffer reader for that aof file every time we read a new array it's going to create a new reader and it just breaks the sync between the files, right? Because you read one array over here and then you create a new reader and then you try to read the next array and it just becomes a mess of things right So instead of creating the reader over here delete this change the argument to R which will be a buff pointer And let's pass in our own reader over here. So in the AOF file, let's say R equals buff.io.newReader, pass in the AOF file over here and pass in the R reader in this V.readArray method call, right? Similarly, in the main.go file up here, let's create in the handleCon function, let's create a reader called buffio.newReader and pass in the connection as the reader and pass the R variable in the readArray function over here. So that's the first thing we need to do. Secondly, we want to fix some issues in our read bulk method over here. We want to basically make this able to return values as well as errors. So instead of having to, you know, basically log all of these errors and do nothing with them, let's return them as well. There we go. And down here, just return nil if there is no error. and handle this over here as such. There we go And over here let just print the error because there not much else we going to do with it and add a continue over here so if we can scan one of the bulk strings we should just continue over to the next one or even not even continue but just break because if you can't scan one then there's no point in scanning the next one because the entire command is going to invalid anyway so let's try to run the server now again and this time it's going to work you can see over here the memory logs right it increases three times from 98 to 191 to 292 so you can already tell just from this that we're loading three keys from the AOF backup if I run the keys command you can see that all three of these keys were now backed up awesome so we fixed all of the issues in our AOF sync function over there now I am going to keep this video short so it's easy for people to watch so I'm going to end the video over here and in the next video after this we're going to continue building eviction policies I believe in the next video we're going to implement the where is it the LRU and the LFU eviction policies for all keys I'll try to do both of these in the next video and then after that we can implement all of the volatile eviction policies as well so stay tuned for that thank you for watching this one like comment subscribe and i'll see you in the next one