Auto-generated transcript Welcome back boys. It's been a while since I sat down and recorded a video. Usually what I do is I just batch record multiple videos in a single setting. So that's pretty helpful. But yeah, it's been a while since I recorded a new video. So we're going to continue building this Go Redis clone. This is going to be part 9 of this series. And in part 8, the previous part, what we did was we built the SplashDB and DBSize commands. right so just a quick review before I move on with the with today's video the DB size command is just going to return the size of the database so like the length of the map inside our database right and the flush DB command will just destroy the entire database and give us a new one right so that's what we built in the previous part of this series in today's video I want to build two things So I want to add first of all checksum protection to the RDB functions over here, right? So that when we save the RDB file, we can use checksums to tell whether all of the database data was stored correctly or not, right? That's the first thing. And the second thing I want to implement is the auth command. So basically the auth command is going to allow us to set password protection on our database, right? And every single client that connects to this server will need to know the password and pass it in to be able to execute commands on the server, right? So these are the two things I want to build in today's video. Let's get started. First, in the RDB file here, we're going to... And let me just expand this as well. down here we're going to write the hash function that we will use to hash different kinds of like byte data this is going to take a reader which will be an io.reader interface there we go and it's going to return a string which will be the actual hash and it's going to return an error in case there's an error otherwise it will be nil right So first of all, let's just create a new hash using the SHA-256 algorithm. Then let's say if and I want to use the io.copy function. What this does is it takes the destination and the source and it will just copy all of the data from the source to the destination. So what we want to do is we want to write all of the data to the hash that we just created, right? and we want to copy this data from the reader that we pass in over here right and this is going to return the number of bytes written and an error message and we only need to concern ourselves with the error over here so let's say if error does not equal to nil then let's let's return the error and also let's just return an empty string. There we go. I don't know why I tried to return error instead of err. And if everything is working perfectly fine then we can actually just return the hex string and the way to do that is to use the hex package and say encode to string and then let's get the hash checksum and you can get the checksum of this hash from the sum function. We'll pass in nil over here and then let's return a nil error as well and that's all we need to do for the hash function. Now up here in the saveRdb function you can see that we're encoding and writing directly to the file over here right using this gob.newEncoder function and passing in the file object directly. So instead of doing this we want to first encode and write the data to a buffer in memory and then we want to get the checksum of that buffer and then we want to write that buffer to the file and then grab the checksum of the file and then just see if both of the checksums from the buffer in memory and the file match right if the if both checksums match then our file was written successfully otherwise there's a mismatch somewhere there's an error somewhere right so let's create a buffer over here I'll give this the type of bytes.buffer and instead of passing in the file object over here we're going to replace this with a pointer to the buffer. So now all of the database data is going to be encoded and written inside this buffer right. So let's grab that data first of all. We're going to go down here and say data equals buff.bytes and this will give us all of the bytes that were written in this in this buffer right then let's grab the checksum of this buffer by saying by calling the hash function that we just wrote and pass in our buffer as a pointer over here and then let's make sure to handle the error as well if there is any let's say log dot print ln rdb um cannot compute buffer checksum and then print the actual error message and then obviously return as well also up here in the error message over here we want to change this to error encoding db because right now it says error saving rdb file right but we're not saving the file over here are we we're just encoding the data to the buffer right so let's change this error message as well just so it means something you know just so it makes sense right let's just say error encoding database and down here now we're ready to write all of the data to the file so let's say error equals F which is the file the RDB file right and run the write method and pass in the buffer bytes over here or actually not the buffer bytes but the data variable over here so let me just explain real quick what I'm doing over here the reason I'm creating a new data variable over here using the buffer bytes instead of just calling buffer dot bytes over here as well is because in the hash function right when this IO dot copy function is going to run and it's going to grab all of the bytes from the buffer right it's going to read all of the bytes from the buffer. The byte cursor is going to be at the end of the buffer and if you try to read from the buffer again it's just going to return an empty file or an empty buffer right because all the bytes have already been read. So what we're doing over here by creating this data variable is we're kind of creating a copy of the buffer in a sense just so we can write it to the file right because this hash function is going to consume the buffer and all of the bytes after that will be empty. If you try to read the buffer again after calling this hash function, it's just going to be empty. So that is why we create a little copy, I guess, of the buffer using this data variable. And we use that to write to the file. Anyway, I hope that makes sense. Let's handle the error as well. Cannot write to file. Let's say that. return the error as well or actually log the error and just return nothing okay so now we have written to the file as well so now the file will have some sort of content right now we can actually um compute the file checksum as well so call the hash function again this time pass in the file let's handle the error as well over here cannot compute file checksum Print the actual error after that and return so now we have both the buffer checksum and the file checksum let's print both of them if they don't match right if the buffer checksum does not equal the file checksum we're going to return an error or log an error I don't know why I confused the two um let's say buffer and buffer and file checksums do not match and instead of using println let's use printf because i want to log the actual checksums as well all right so after this add a new line and let's say the file and print a string over here which will be the file checksum add another new line then print the buffer which will be the buffer checksum over here and then finally another new line so this will basically just show us both of the checksums and we can like compare them and see if they actually don't match or whatever is going on right if they don't match just return otherwise we print this success message we've saved the rdb file and that is all we need to do Awesome, right? So that's all we need to do to implement checksum protection in rdb files. Now, let's test this out. First of all, let me just shut down the previous Redis server, if it's still running. There we go. Let's run this server. And open up the client over here. let's run the save command and you can see they don't match as of right now and we log both of the checksums over here. So there's basically two more issues that we need to solve. First of all let me just explain everything. First of all we need to make sure that the data that we write to the file is actually being flushed out to the file right on disk and not stored in some memory buffer over here right what the operating system often does is when you try to write to a file it basically stores whatever you write inside a memory buffer and it just writes that buffer inside the file later on whenever it decides to right we want to force it to write to the file just in case right just in case the data isn't being written to the file we want to just make sure that we are actually writing to the file and you can do that with this sync function so sync is going to just as it says commit the current contents of the file to stable storage right flushing the file systems in memory copy so the the data that we write to the file which is being stored inside some memory buffer managed by the operating system it's going to write that to the disk right to the file on disk so make sure to call this this is going to return an error right it's going to return an error and we can just handle that over here make sure that the error is not nil over here and if so print the error cannot flush file to this log the error after that and return so this is going to make sure that whatever we write we write to the file right it's going to be committed to the disk right but there's actually one more crucial step that we need to take i mentioned up here the reason why we create this data variable and create a copy of the bytes right is because this hash function is going to consume the buffer and move the byte cursor to the end so that if we try to read from the buffer again it's just going to return empty bytes right which is why we needed to create a copy of the buffer over here, right? We doing something similar in the file over here right In the file on disk So when we write to the file right We're going to write a bunch of bytes over here. Let me just, I don't know, create a new file over here. When we write to the RDB file, we're going to just fill it up with a bunch of bytes, right? The cursor, the byte cursor is going to be at the end of the file over here, right? And at the end of the file you can see there's no like data over here after this right all the data is before the cursor So what we need to do is we need to reset the cursor so that And let me close this so that when we hash it again And when the hash tries to read the file it reads from the beginning and not from the end or some in-between like location, right So in order to do that, and I hope I'm making sense over here I feel like I'm just rambling, but hopefully I'm making sense and hopefully you can understand what I'm saying over here But what we basically need to do now is we need to use the F dot seek function pass in 0 over here as the offset and pass in IO dot seek start what this is going to do is it's going to move the byte cursor all the way back to the starting of the file right to the beginning of the file so that when we try to hash it and when the hash function tries to read the file it will read from the beginning and not the end all right now this is going to return two values we're only concerned with the error value so let's just say if underscore error is this function call and error does not equal to nil then we basically print the error message over here and say cannot seek file and we return that's all now with all of that done let's try running the go server again run the client run the save command and you can see this time it actually works the checksum protection is working as well both of the checksums match and we successfully saved the rdb file And if I open it up, it's going to just have a bunch of data over here like this. And yeah, that's all we need to do to implement checksum protection in RDB files. Now, if we ever have like a system crash or a server crash or something like some sort of like intrusion in between, right? And the data from the buffer isn't written to the file correctly, we're going to return an error for that. and we will know whether the rdb snapshot was successfully saved or not so this is kind of like an extra step to just make this whole program much more like much more stable and reliable right because you know that if the if the checksums don't match then something has gone wrong right previously we didn't have this kind of protection right but now with this checksum protection we have that kind of like reliability in this server so that's all done uh the other thing i want to do is implement the authentication but i think this video is getting a bit too long already and i'm trying to record shorter videos just because i think that those are much more easier for people to watch uh and this video is already getting too long we've been recording for like 20 minutes even if I cut some of the you know the silence and the mistakes out it's still going to be like a 15 minute video at least which I think is long enough I did say I'm going to implement the authentication stuff in this video but it's just getting too long and I want to record shorter videos so we'll do the authentication stuff and the auth command and everything right this command in the next part of the series so stay tuned for that thank you so much for watching this part and I will see you in the next one you