Syncing projects with Google Drive - Part 10 (New sync algorithm)

Testing, testing, testing. One million, two million, three million. Testing! All right, back in the DriveSyncer project. In the previous video, we built the progress bar functionality right here. We basically made all of the uploads concurrent, and we created these progress bars over here so you could actually see the upload progress. Right now, the next major feature I want to implement is a much more better and performance sync algorithm. Basically, let me just show you. Where is the sync API over here? Yeah, here we go. Okay, so basically, when we upload a project folder right now, what we do is we delete the previous folder completely, and we upload the entire folder again, right? And this is, it works, it gets the job done. And to be fair, I don't think you're gonna run into any Google Drive limits over here, because their API limits are actually very generous. But at the same time, I just don't like this because it's really wasteful, right? You have to upload every single file all over again every time you try to sync a new project, right? Or even an old project, right? Which is not good, right? So what I want to do is, instead of deleting the previous folder, I want to implement a much better upload algorithm that can basically just skip the files that have not been changed and only re-upload the changed files, right? The files that we've actually modified, right? And that's a lot of work, right? It's gonna take about three or four videos to actually implement that entire thing. So stick around, and we'll get started on that in this video. Before we do that, though, I want to fix one feature in the progress bar that we built last time. So let me just run a test upload over here so you can see what I mean, right? Let's just try uploading something right now. By the way, let me also zoom in so it's easier for you guys to see. Okay, so it needs to grant access, so I'm gonna go do that real quick. Okay, here's my code. Why didn't that work? Okay, so there's still some issue with the OAuth implementation where it just exits, crashes the first time, but it runs the second time. And there we go, it's actually uploading now. Okay, so there's these two projects that we're uploading right now. I'm gonna let them finish, and then we see what the problem is. Basically, the progress bar doesn't hit 100, right? And Google is telling me that some hacker logged in. Yeah, that was me. Basically, these progress bars are not going to hit 100% progress, right? Even though the upload finishes, it doesn't wait for the progress bar to reach 100%, and so it looks like it didn't finish properly, right? So that's what we need to fix now. And there you go. See, the GoRedis video reaches 100%, but this DriveSyncer video project, this only reaches 94.44%, and it looks like it's unfinished, right? Even though the entire project was uploaded successfully, it looks like it was unfinished. So we're going to fix that, and the solution to this is extremely simple, extremely simple. We just need to double the size of our wait group over here. So we have this sync.waitGroup object over here, and its length is the number of folders that we're uploading, right? Let's double this. Let's just say multiply it by two. And what we're going to do now is also call wg.Done in this goroutine that handles rendering the progress bars. So here's what's going to happen, right? It's gonna upload the folder to Google Drive, and then it's gonna say wg, and it also gonna render the progress bar all the way to 100, and then it gonna call wg.Done. And so both of these operations are going to complete, and the program isn't going to, like, shut down prematurely anymore. This is all we need to do. Now let's just test this out. Run the entire project again. Actually, by the way, before we run this, I do want to add one more thing. Let's go to the main.go file over here, and right up top, let's just add an empty line over here. Let's just say fmt.Println, and we don't need to print anything. I just want to have a new line over here. Why? Because you see over here, it just started the upload process and it erased the previous line over here, right? And that just looks weird. This whole thing looks like one single command, right? Even though it was two commands, it just erased the line above, and that just makes it look weird. So to fix that, I'm just adding an extra line over here. So that's the one it's going to erase. This is what it's going to look like, actually. If I run this, huh? Okay, that was not supposed to happen. It just erased my terminal line again. That was not supposed to happen. Okay, so apparently I'm not so good at this stuff. But the progress bar is going to reach 100% in this case. Hopefully. Hopefully I didn't mess that part up. Let me just move that down a bit and see what the hell I did wrong. Yeah, there we go. See, both of the upload progress bars reach 100% now. So that issue has been fixed completely. Awesome. Okay, so with that fixed, let's move on to building the new upload algorithm that I was talking about in the beginning of the video. To do this, we first need to figure out which files are actually changed and which are unchanged, right? We're going to upload the files that actually changed, and we're going to ignore the files that are not changed, right? And so we need some sort of a tracker to figure out which files we're supposed to upload and which files we don't need to upload. So inside this API package, let's create a new file and call it tracker.go. It's gonna have package API, and it's gonna implement a bunch of functions. First of all, let's create a struct. How do you create a struct again? Type Tracker struct. There we go. I have such a short memory, guys. It's gonna take a couple of fields. First of all, the base folder ID. This, in our case, is just the programming folder where we storing all of our projects, right? It's going to be a string. It's also going to take the Drive, which is a pointer to models.Drive. And it's going to take the folder, which is a models.Folder pointer. Also, it's gonna take the folder ID, which is gonna be a string, and this is the ID of the folder on Google Drive, right? This is the base folder ID on Google Drive, and this is the project folder ID on Google Drive, right? Finally, it's gonna take a bunch of nodes, and this is gonna be a map. All right, it's gonna be a map to another struct that we'll create down here. I'll call this node struct. There we go. So we don't actually need to implement this struct right now because we're not going to use it right now. This is going to be in a future video, probably the next video after this. For now, let's just finish the tracker struct. This nodes key is going to be a map of strings and node pointers. Basically, this node struct is going to track every single file changes inside that folder, and we're going to use this to figure out which folder we're supposed to upload and which ones we're supposed to not upload. Now, let's create a new tracker function over here, new tracker, and this will take the drive, which is models.drive pointer, the folder, which is a pointer to models.folder, and it's going to take the base folder ID, which is going to be a string, and it's going to return a pointer to the tracker object. This tracker will be a tracker will assign the parameters over here. DRV, that's the drive. This is the folder, and then the base folder ID. There we go. And then it's going to return a reference to that tracker. There we go. Now, let's actually start tracking the changes, right? The very first change that we need to track is whether the project itself has been uploaded before or not, right? And we can tell this by checking whether the folder of that project exists inside our base folder or not. If it does not exist, then that means that we've never uploaded this project before and that we should just upload every single file, right? We don't need to add any more checks after that because if the project folder doesn't exist at all, then we know that the entire project needs to be uploaded, right? So we can skip the checks after that. So let's create a function over here. This will be a pointer receiver for the tracker struct. We'll call this project exists. It doesn't need any arguments, and it's going to return a Boolean and an error message. Now, quick recall, the way we search for files in the Google Drive API is to first write a query, right? This is what we've done all this time, right? So let's do fmt.Sprintf. Let's say MIME type is gonna be a string over here, and that string is going to be the folder MIME type so that we're actually looking for folders, right? And percentage as in parents, and this is going to be the base folder ID, if I can type that out over here. Tracker.base folder ID. There we go. And trash equals false, right? We don't want to search for folders that are in the trash folder, right? Now, let's do if response and error equals t, which is a tracker.drive.api.files.list, and then list. A lot of stuff over here. Dot query, and here we'll pass in our query. Dot fields, and here we'll pass in, we basically need two fields over here, right? The ID and the name, just so we can match this folder and see if it exists, right? Dot page size, which is going to be one because we only expect this folder to, like, we only expect one folder with this name and these parameters, right? If there are multiple matches, then that probably means we fucked up somewhere. Ideally, there should only be one folder with this name inside this folder. And now let's run this API call by saying do, and then error equals nil, and there we go. Now, do note that I'm using error equals nil over here, not error does not equal to nil. In most of the Go code that you read, you're gonna read error does not equal to nil, right? That means that an error exists, right? If you actually focus on the conditional over here, error equals to nil, that means there was no error, right? So what I'm basically saying over here is run this API call, and if there was no error, then do all of the stuff I'm going to write over here, right? Now, we need a switch statement. So switch length of our dot files is going to be our condition. First case, zero, which basically means that there is no folder in our base folder that matches this project name, right? That means we can just return false. The project does not exist. And we can return nil for the error. This means that we're going to upload the entire project all over again, right? Case one, and here we can return true and nil. One more thing we can also do over here is we can assign the folder ID in the tracker object over here. So we can say r dot files and select the first element in the files and say ID, and then, um, a default case over here. And if it's not zero and it's not one, then that means that it's larger than one, right? Multiple matches found. So here let's return an error. Let's return, I guess, false for the Boolean value, and let's say errors.new, multiple project folders found. And by the way, the only reason I'm adding this default check over here is on the 0.001 chance that the Google API returns some weird error over here, right? Because this is impossible to reach. In fact, I actually comment this out as well. Impossible to reach because of page size one. We added this page size one call inside of our, inside this, this entire API call where we're calling the Google Drive API. We added this page size equals one parameter as well, right? So it is impossible to get more than one files in the response, right? So this default case is largely useless. It's not going to ever execute unless the Google Drive API breaks and it just, you know, starts returning gibberish, right? So in most cases, this should not even run, right? In most cases, you would just run case zero and case one. Anyway, so that's all if the error was nil, right? Now, if there was an error in this API call, then we'll just return false and return the error. Let's say that's all we need to do for this one. Now, one final function that we're going to implement is the get checksum function. So let's say `func (tracker)` — this is another method receiver. We call this `getChecksum`, and it will take an OS object parameter, or object pointer, and it will return a string, which will be the hash checksum of the file, and an error message if there is one. Right, let's say `hash := sha256.New()`. SHA-256 is the hashing algorithm that we're going to use. And then let's say `if _, err := io.Copy` and then let's copy to the hash from the file that we have open, and say error does not equal to nil, then we should just return an empty string over here along with the error message, right? Otherwise, let's grab the hash, the hash checksum over here, by saying `hash.Sum` and pass in nil over here because we don't need to append anything. And then we also need to convert this into a string because this is not gonna be a string at first. It's gonna be a byte slice. And so, to convert this to a string, we'll say `hex.EncodeToString`. I believe that's what you say. Yeah, there we go. Encode to string and pass in a bunch of bytes over here, like this sum variable, and then return nil for the error. And that's all we need to do. So these are the three functions that we're going to actually use in the tracker right now. We're gonna implement a couple more in future videos, but for now, this is good enough for us. Let's actually implement this. Let's integrate this in our API, in our upload algorithm, by going to the `api/sync.go` file over here. First thing I'm gonna do is I'm just gonna comment out the entire delete previous folder function over here. Or, in fact, not comment this out, but actually go all the way up to the upload folder function over here, and this is where we're actually calling the delete previous folder function. So let's comment this out. And if I do that, if I comment this out, then we're not going to delete the previous folder, and we're going to have a bunch of duplicates in our projects, right? Which is fine for now. We're going to fix this in future videos. But now we do need to fix a couple things. First of all, let's go down to the `filepath.WalkDir` function. You know that this WalkDir function is first of all going to create the project directory, right? Because that's the first path that this function is going to access, right? It's going to be the root path of this project. So we can say something like, if the current path that we're on matches the folder's path, the project folder that we're uploading, then we can do a couple of things here. First of all, let's just go up over here and just below the subfolders, let's create a tracker object called `newTracker` and passing the parameters over here. Let's pass in drive, the folder, and the base folder ID, which is going to be `base.ID`. There we go. And then let's see if the project already exists or not. So I'll say `projectExists` and an error message over here, and call `tracker.ProjectExists`, right? And this will return a boolean and an error, just like we implemented. And first of all, let's just handle the error before we do anything else. I'll say `log.Fatalf` to crash the entire program. I'll print the folder name first and foremost. We'll say `folder.Name` over here as the first argument, checking if project exists already, and then we'll print the error message along with a new line. So let's add the error message over here, and that's all we need to do. Cool. Now let's go back down over here in the `filepath.WalkDir`. First of all, if the project already exists, then I want to, let's just add a log first of all: "Project folder already exists, assigning ID." So how we implemented subfolders over here, right? Where is it? We basically add the ID of the subfolder in that subfolders map over here, right? So this happens automatically every time we create the subfolder. But since we're not deleting the project folder anymore, it's going to exist already. We need to make sure that we actually assign its ID to the subfolders list, right? So I can say over here, `subfolders`, and the key will be the `folder.Path`, the project folder's path, and the ID will be `tracker.folderID`, right? And this will get assigned as soon as we call the `ProjectExists` function over here. It's going to assign this `t.folderID` value over here once it finds the project, right? And so we can just use that. And if the project folder does not exist already, then we should create the project folder. And by the way, before anything else, let's also return nil over here because there was no error, right? And let's go up outside of this function and write the create project folder function. This is just going to be create project folder. There we go. It's going to take the drive, it's going to be models.drive, the base folder ID, which is going to be a string, and the name, which is going to be a string, right? The project folder name. And it will return a drive pointer and an error message. And this is going to be a very short function, literally just one line. It going to return create subfolder, the function that we implemented several videos ago. We'll pass in the drive, the base folder ID. This will be the parent, of course, and then the d over here. We need to pass in the fs.dir entry. I'm just going to change this because I only have a name over here, so that's what we're going to use. Let's go to the subfolder function down here. Instead of using this fs.dir entry, let's just use the folder of the name because we literally only need the name of the folder over here and nothing else, right? So let's just use that, and let's also fix the error that this is going to cause in our code over here. There we go. And over here, in the where we're actually calling the create subfolder function, just use d.name instead of name, right? Pass in the actual name of the folder. Now that we've done that, let's implement this. Let's use this create project folder function that we just implemented. I'll say response and error equals create project folder. Pass in drive, there we go, and then base.ID, and also the name, which will be folder.name, I guess. If there's an error, if error does not equal to nil, let's say logs.fatalf again, "Fail to create project folder," and then print the error message along with a new line. There we go. And of course, make sure to assign the ID of this folder, the Google Drive ID of this folder, to the subfolders map that we're using over here so that we can actually upload all of the files inside this folder, right? So let's say subfolders. I'm just gonna keep that in the video. subfolders, folder.path, and response.ID. There we go. And return nil so we don't execute all of the other code below this, right? And that should be all we need to do. Now, this is what my Google Drive looks like right now. This DriveSinker video project over here that we're actually uploading. And let's actually run the entire upload process again. Let's see, which folders am I uploading? Okay, these two. Let's run this again and see what happens, see if the stuff works or not. Now, spoiler alert, I don't think most of this is going to work. I'm pretty sure we just broke the entire upload process. We're going to fix this in future videos. It's going to take, like I said in the beginning, guys, this is going to take like two or three videos to actually implement the entire algorithm for synchronizing all of this stuff. And it looks like it just failed already. Awesome. Let me try running this again. I don't know what happened over there. Okay, so there's some errors in the log file over here, but I don't understand them one bit. So I'm just gonna delete these folders over here. Instead of actually trying to fix the... I'm just gonna empty my entire Google Drive, and then we're going to run the same process again, and it still doesn't work. Nice. Okay, so it's failing to find the parent folder. Where is this error message? Let's do a quick search. Okay, right here. Fail to find error. Fail to find parent folder. Okay, so this happens over here. Okay, and it's failing to find the project folder. Okay, we're gonna do a bit of live debugging over here. Okay, guys, before I actually try to solve this error, let me go to main.go, and let's not redirect all of our logs to a log file, since that's just... since I want to see them in the terminal now whilst I'm debugging this entire application. And also, in sync, in the command over here, let's not call render bars because I don't want to see the progress bars anymore, not while I'm debugging this application. So let's run this again, and it's still gonna fail, but at least it's going to show us the logs, right? So it says project folder already exists, and it's failing to find that print folder. Most likely, it's just not assigning the ID properly. So let's go fix that. Oh, wait, wait, wait, wait. I figured it out. Oh my goodness. So the problem is that I'm using folder.path over here in the subfolders list over here as a key. I'm using the folder.path instead of folder.name. This is supposed to be the folder.name. That's how we filter. That's how we map the subfolders by their by the folder name, not by folder path. Oh my goodness, I can't believe I've messed all of that up. Let's run this again, and this time it's going to actually work. I can't believe I wasted like 10 minutes trying to debug this application. Yeah, this time it's working. Awesome. Oh my goodness, how long is this video again? Yeah, see, we're approaching 40 minutes of recording again. I try so hard to not make such long videos, but then the projects I build are also so complicated, and all this stuff happens, right? So hopefully you have the attention span to watch this entire video because I'm trying my best, but the videos just get so long, you know? Anyway, this entire process finished, and here are the project folders. Why is there nothing in here? Over here. Okay, there we go. Oh, okay. All right, so as you can see, there's a ton duplicates over here, right? Every single file got uploaded like twice. This is because we're not deleting the inside folder anymore, right? So the upload process works, but it's a lot more broken and messed up now, right? In future videos, or in the next video probably, we're going to fix this and track every single files changes so that we only upload the files that have been changed, and we delete any other files, right? We will leave the unmodified files unchanged. We upload the new files and the change files, and any single file that we delete from our local folder, we should delete that from Google Drive as well, right? So we're going to implement all of that stuff in the next video or the one after that. For now, we're done with this video. We've built the a basic tracker over here, and we've implemented, we've integrated with our upload algorithm. Then I'm gonna stop the video there, guys, because this is getting extremely long. 40 minutes just of recording, and then I have to edit this entire thing as well. It's gonna take forever. So I'm gonna stop over there. Hopefully you learned something new over here because this project is starting to become like a really great useful project, especially for me. Like, the only reason I'm even building this is because I personally am going to use this because I'm the kind of guy who, especially for personal side projects, I just forget to push everything to GitHub, and I just, you know, like, just code locally and just forget to push it to GitHub, especially when it's a personal project, especially when it's a side project that I'm just working on that nobody else is going to really see. I just forget to upload all of those things to Google to GitHub. So with this project, this DriveSinker project that I'm building, I can just automate that entire process for me, right? Even if I don't upload to GitHub, this program can just run automatically and synchronize changes with my programming projects to Google Drive, right? So that's going to be awesome. This program is going to be awesome once it's finished. But yeah, thanks for watching. I'm going to stop the recording now. See ya.