Building a Redis clone in Go - Part 9 (RDB Checksum Protection)

Welcome back, boys. It's been a while since I sat down and recorded a video. Usually, what I do is I just batch record multiple videos in a single setting. So that's pretty helpful. But yeah, it's been a while since I recorded a new video. So we're going to continue building this Go Redis clone. This is going to be part 9 of this series. And in part 8, the previous part, what we did was we built the FlushDB and DBSize commands. Right, so just a quick review before I move on with today's video. The DBSize command is just going to return the size of the database, so, like, the length of the map inside our database, right? And the FlushDB command will just destroy the entire database and give us a new one, right? So that's what we built in the previous part of this series. In today's video, I want to build two things. So I want to add, first of all, checksum protection to the RDB functions over here, right? So that when we save the RDB file, we can use checksums to tell whether all of the database data was stored correctly or not, right? That's the first thing. And the second thing I want to implement is the AUTH command. So basically, the AUTH command is going to allow us to set password protection on our database, right? And every single client that connects to this server will need to know the password and pass it in to be able to execute commands on the server, right? So these are the two things I want to build in today's video. Let's get started. First, in the RDB file here, we're going to... And let me just expand this as well. Down here, we're going to write the hash function that we will use to hash different kinds of, like, byte data. This is going to take a reader, which will be an io.Reader interface. There we go. And it's going to return a string, which will be the actual hash, and it's going to return an error in case there's an error. Otherwise, it will be nil, right? So first of all, let's just create a new hash using the SHA-256 algorithm. Then let's say if, and I want to use the io.Copy function. What this does is it takes a destination and the source, and it will just copy all of the data from the source to the destination. So what we want to do is we want to write all of the data to the hash that we just created, right? And we want to copy this data from the reader that we pass in over here, right? And this is going to return the number of bytes written and an error message, and we only need to concern ourselves with the error over here. So let's say if error does not equal to nil, then let's return the error and also let's just return an empty string. There we go. I don't know why I tried to return error instead of err. And if everything is working perfectly fine, then we can actually just return the hex string. And the way to do that is to use the hex package and say EncodeToString, and then let's get the hash checksum. And you can get the checksum of this hash from the Sum function. We'll pass in nil over here, and then let's return a nil error as well. And that's all we need to do for the hash function. Now, up here in the saveRDB function, you can see that we're encoding and writing directly to the file over here, right? Using this gob.NewEncoder function and passing in the file object directly. So instead of doing this, we want to first encode and write the data to a buffer in memory, and then we want to get the checksum of that buffer, and then we want to write that buffer to the file, and then grab the checksum of the file, and then just see if both of the checksums from the buffer in memory and the file match, right? If both checksums match, then our file was written successfully. Otherwise, there's a mismatch somewhere. There's an error somewhere, right? So let's create a buffer over here. I'll give this the type of bytes.Buffer, and instead of passing in the file object over here, we're going to replace this with a pointer to the buffer. So now all of the database data is going to be encoded and written inside this buffer, right? So let's grab that data. First of all, we're gonna go down here and say data equals buff.Bytes, and this will give us all of the bytes that were written in this buffer, right? Then let's grab the checksum of this buffer by saying, by calling the hash function that we just wrote, and pass in our buffer as a pointer over here. And then let's make sure to handle the error as well, if there is any. Let's say log.Println("Cannot compute buffer checksum") and then print the actual error message, and then obviously return as well. Also, up here in the error message over here, we want to change this to "error encoding DB" because right now it says "error saving our DB file", right? But we're not saving the file over here, are we? We're just encoding the data to the buffer, right? So let's change this error message as well, just so it means something, you know, just so it makes sense, right? Let's just say "error encoding database". And down here, now we're ready to write all of the data to the file. So let's say error equals F, which is the file, the RDB file, right? And run the Write method and pass in the buffer bytes over here. Or actually, not the buffer bytes, but the data variable over here. So let me just explain real quick what I'm doing over here. The reason I'm creating a new data variable over here using the buffer.Bytes instead of just calling buffer.Bytes over here as well is because in the hash function, right, when this io.Copy function is going to run and it's going to grab all of the bytes from the buffer, right? It's going to read all of the bytes from the buffer. The byte cursor is going to be at the end of the buffer, and if you try to read from the buffer again, it's just going to return an empty file or an empty buffer, right? Because all the bytes have already been read. So what we're doing over here by creating this data variable is we're kind of creating a copy of the buffer in a sense, just so we can write it to the file, right? Because this hash function is going to consume the buffer, and all of the bytes after that will be empty. If you try to read the buffer again after calling this hash function, it's just going to be empty. So that is why we create a little copy, I guess, of the buffer using this data variable, and we use that to write to the file. Anyway, I hope that makes sense. Let's handle the error as well. "Cannot write to file." Let's say that. Return the error as well. Or actually, log the error and just return nothing. Okay, so now we have written to the file as well. So now the file will have some sort of content, right? Now we can actually compute the file checksum as well. So call the hash function again. This time, pass in the file. Let's handle the error as well over here: cannot compute file checksum. Print the actual error after that and return. So now we have both the buffer checksum and the file checksum. So let's print both of them if they don't match, right? If the buffer checksum does not equal the file checksum, we're going to return an error or log an error. I don't know why I confused the two. Let's say, "Buffer and file checksums do not match." And instead of using Println, let's use Printf because I want to log the actual checksums as well. All right, so after this, add a new line and let's say the file, and print a string over here which will be the file checksum. Add another new line, then print the buffer, which will be the buffer checksum over here, and then finally another new line. So this will basically just show us both of the checksums, and we can, like, compare them and see if they actually don't match or whatever is going on, right? If they don't match, just return. Otherwise, we print this success message: we've saved the RDB file, and that is all we need to do. Awesome, right? So that's all we need to do to implement checksum protection in RDB files. Now let's test this out. First of all, let me just shut down the previous Redis server if it's still running. There we go. Let's run this server and open up the client over here. Let's run the SAVE command, and you can see they don't match as of right now, and we log both of the checksums over here. So there's basically two more issues that we need to solve. First of all, let me just explain everything. First of all, we need to make sure that the data that we write to the file is actually being flushed out to the file, right, on disk, and not stored in some memory buffer over here, right? What the operating system often does is, when you try to write to a file, it basically stores whatever you write inside a memory buffer. And it just writes that buffer inside the file later on whenever it decides to, right? We want to force it to write to the file just in case, right? Just in case the data isn't being written to the file. We want to just make sure that we are actually writing to the file. And you can do that with this Sync function. So Sync is going to, just as it says, commit the current contents of the file to stable storage, right? Flushing the file system's in-memory copy. So the data that we write to the file, which is being stored inside some memory buffer managed by the operating system, it's going to write that to the disk, right, to the file on disk. So make sure to call this. This is going to return an error, right? It's going to return an error, and we can just handle that over here. Make sure that the error is not nil over here, and if so, print the error: cannot flush file to disk. Log the error after that and return. So this is going to make sure that whatever we write, we write to the file, right? It's going to be committed to the disk, right? But there's actually one more crucial step that we need to take. I mentioned up here the reason why we create this data variable and create a copy of the bytes, right, is because this hash function is going to consume the buffer and move the byte cursor to the end, so that if we try to read from the buffer again, it's just going to return empty bytes, right? Which is why we needed to create a copy of the buffer over here, right? We're doing something similar in the file over here, right? In the file on disk. So when we write to the file, right, we're going to write a bunch of bytes over here. Let me just, I don't know, create a new file over here. When we write to the RDB file, we're going to just fill it up with a bunch of bytes, right? The cursor, the byte cursor, is going to be at the end of the file over here, right? And at the end of the file, you can see there's no, like, data over here after this, right? All the data is before the cursor. So what we need to do is we need to reset the cursor, so that... And let me close this, so that when we hash it again, and when the hash tries to read the file, it reads from the beginning and not from the end or some in-between, like, location, right? So in order to do that, and I hope I'm making sense over here. I feel like I'm just rambling, but hopefully I'm making sense, and hopefully you can understand what I'm saying over here. But what we basically need to do now is we need to use the `f.Seek` function, pass in 0 over here as the offset, and pass in `io.SeekStart`. What this is going to do is it's going to move the byte cursor all the way back to the starting of the file, right, to the beginning of the file, so that when we try to hash it and when the hash function tries to read the file, it will read from the beginning and not the end. All right, now this is going to return two values. We're only concerned with the error value, so let's just say if underscore error is this function call and error does not equal to nil, then we basically print the error message over here and say, "Cannot seek file," and we return. That's all. Now, with all of that done, let's try running the Go server again. Run the client, run the SAVE command, and you can see this time it actually works. The checksum protection is working as well. Both of the checksums match, and we successfully save the RDB file. And if I open it up, it's going to just have a bunch of data over here like this. And yeah, that's all we need to do to implement checksum protection in RDB files. Now, if we ever have a, like, a system crash or a server crash or something, some sort of, like, intrusion in between, right, and the data from the buffer isn't written to the file correctly, we're going to return an error for that, and we will know whether the RDB snapshot was successfully saved or not. So this is kind of like an extra step to just make this whole program much more, like, much more stable and reliable, right? Because you know that if the checksums don't match, then something has gone wrong, right? Previously, we didn't have this kind of protection, right? But now, with this checksum protection, we have that kind of, like, reliability in this server. So that's all done. The other thing I want to do is implement the authentication, but I think this video is getting a bit too long already, and I'm trying to record shorter videos just because I think that those are much more easier for people to watch. And this video is already getting too long. We've been recording for, like, 20 minutes. Even if I cut some of the, you know, the silence and the mistakes out, it's still gonna be, like, a 15-minute video at least, which I think is long enough. I did say I'm going to implement the authentication stuff in this video, but it's just getting too long, and I want to record shorter videos. So we'll do the authentication stuff and the AUTH command and everything, right, this command in the next part of the series. So stay tuned for that. Thank you so much for watching this part, and I will see you in the next one.