Syncing projects with Google Drive - Part 12 (Ignoring unchanged files)

All right, back in the DriveSinker project. We've been building the tracker functionality over here for the last couple of videos. I think for the last two videos we've been working on this tracker struct and its associated methods and everything. Just a quick recap: We want to use this tracker struct to track every single file that we upload to Google Drive and then to see which files have been modified, which have not been modified, which have been unchanged, which need to be deleted, and on and so forth. Track all of those things and then use that to basically only upload the files that have been changed. Right now, we're uploading every single file and folder and just deleting everything that was previously uploaded. Right, that is a lot of wasted bandwidth and uploads, so we're going to fix that by using this tracker and building a much more efficient sync algorithm. So in the previous video, we built the walk project tree function over here, and what this basically does is it will recursively, recursively walk through every single folder in this project on Google Drive. So all of the remote files and folders that we have, it's going to walk through all of them recursively and build a list of nodes over here, a map of nodes that we can then use to compare the remote project to our local project on our local machine. So that's what we've done so far. We're going to continue building this tracker now and build a couple more functions, integrate them in our algorithm, and hopefully we can finish the entire thing today. Let's see. So the first thing we need to build right now is a should upload function. I want to have a function that tells me whether I should upload a given file or not. And this will, of course, be a method receiver, just like everything else we've built. So call this should upload. It'll take a path, which will be a string, and then it will return a boolean, which will tell us whether we should upload the file or not, and also an error message if there is an error. Now, the first thing we need to do is actually see if this path exists in our nodes, right? So for that, we can do something like if node, okay, and then assign that to t.nodes and try to get the path from over there. And if this is okay, if this node exists, this means basically that this path exists on Google Drive, right? Because these nodes are coming from the walk project tree function, where we create this remote and local path mapping, right? So if this is okay, that means that this path, that this given path, is going to exist on Google Drive. Now, there's two things we need to do over here to actually decide whether to upload this node or not. Basically, first of all, this node could be a folder or it could be a file. If it is a folder, then our job is a lot easier because we can just say return false and nil. Basically, don't re-upload this because if it's a folder, it's just a directory with a name on it, right? Folders on their own don't really do anything. They don't have anything. They're just empty containers, right? So if the node already exists and it's a folder, that means that the folder already exists and we don't need to do anything special for it, right? So we can just return false and nil for the error. Now, if this node is not a folder, if it is a file, then we actually need to do some work over here. So first of all, let's just open that file on our local machine by using os.open, passing the path over here, and then, of course, handle the error. We don't want to ignore any errors. I'll just return false and then the error, and also probably a good idea to close the file using defer once we're done with. Now we want to compare the checksums of both files: the checksum of the file on Google Drive and also the checksum of the other file on our local machine. And if both of those don't match, that means there that the file has been modified, right? And we should upload it again. So let's first of all get the local checksum and the error by running t dot get checksum. This is this function below over here that we defined in, like, I guess, two, three videos ago. We're gonna pass in the file over here. Make sure to handle the error. Just return it, I guess, and then you can just compare both checksum. So we can say something like return local checksum does not equal to the node.checksum. This will be the checksum of the node on Google Drive, of the file on Google Drive. And if this doesn't match, then this conditional will return true. Basically, that means that if the checksums of both files don't match, then we should re-upload that file. And we can return nil for the error since there is no error in this case. Now, you'll notice that I'm still getting an error over here because I'm not handling this return conditional over here. If the node exists, then we do all of these things, right? But if this node doesn't exist on Google Drive, that means it's a completely new file, then we should obviously upload it. So we'll say return true, and this is kind of like hard coded over here, but it makes sense, right? All right, now let's actually integrate this into our sync algorithm. And to do that, we can just go over to api_sync.go. And I just realized I haven't zoomed in over here, so if you can't see this, I'm sorry. I'm going to zoom in right now, and that should be a lot easier to see. So in the file_path.walk_dir block over here, down here, after we've ignored all of the files to ignore, we can say shouldUpload and error equals tracker.shouldUpload and pass in the path. And before anything else, let's handle the error as well. Let's just say something simple over here. Let's say print the folder name first and say cannot upload path. Whoops! And then we can just print the error message over here with a new line. Don't forget the new line. And we can return the error message over here. Now, if this says if this should upload is false, which we can check like this, then that means we shouldn't upload the file, right? So let's add a log for that as well. Let's just say skipping this path and let's mention that we're doing that because it's unchanged, and of course, a new line character as well. And in this case, we can return nil. No errors, right? We're just skipping the file because it's unchanged. Now I'm gonna empty my programming folder over here so we can start fresh, and then let's see if this works. The first time I run this, it should just upload every single file because everything is empty over here in our Google Drive, right? But the second time I run this program, it should skip most of the files because they're already uploaded, right? So let's see if that works. All right, so that finished, and here is my folder, and it has all of the files from this project. Now let's try uploading the same thing again. So let me clear this, and then let's run this again. You can see it says project already exists, and it's walking the project tree now. And here are all of the files on Google Drive, and it literally just skipped uploading every single file over here because they're all unchanged. Now there is one major, major issue we need to fix over here. You can see it created this subfolder API in parent auth. Let me show you what that looks like on our local machine first. Basically, I added this subfolder over here just to show you an error that we have to fix over here. You can see that we have an API folder up here that has the sync.go and tracker.go files, right? Well, I also added an API folder inside auth, and the only thing it has is this hello.txt file that just doesn have anything, just a bunch of random text, right? Now let me show you what happens on Google Drive. We have this API folder right next to our project for some reason, and inside our project, inside auth, there is no API folder. There is no hello.txt file. That stuff got uploaded over here, and that makes no sense, right? Now, the reason this is happening is this is very interesting. Follow me. If you go over to sync.go, you'll see that in the subfolders map over here, we're using the name of the subfolder instead of the path of the subfolder, and that is a huge mistake because that means that if you have a couple of subfolders in your project that have the same name, they're gonna duplicate over here, right? They're gonna duplicate, and then you have a problem because you have no idea how to tell which subfolder is which, right? Now, the simplest solution over here that I found is to use subfolder paths instead of subfolder names, and that will just make sure that every subfolder has its own unique path. Even if they're, even if they have the same name, they would have a different path because they're in a different path in your system, right? So let's fix that as well. I'm going to delete everything over here again, just because I like to start fresh, and then let's fix this. Now, the very first thing, we can't use the base folder over here because we're using file paths, right? Since we're using folder paths over here, we won't be able to use this, the base folder over here, because it doesn't have a path. It's on Google Drive already, right? So we're just gonna have to remove that and just have an empty map over here. Now this should be fine because we only use the base folder to actually create the root project folder, right? Which we're already doing in this function over here, in create project folder, and we're already passing the base folder ID and everything, so nothing should break by removing the base folder from the subfolders map. And now let's start using folder paths instead of folder names. So down here in subfolders, everywhere you use folder.name as the key, just start using folder.path. This is super simple. So in both of these locations, let's use folder.path as the key for the subfolders map. Let's keep going down, and over here as well, we want to use the folder name. No, we know we want to use the folder path instead of the folder name. So over here, and in the subfolder sub name over here, we can use the path instead of using the sub name because the path is already going to be the subfolder name in this case, right? Same thing down here as well. Use subfolders path over here instead of the sub name. And over here, the parent path, we want to remove the call to file_path.base because that's going to remove our path, and we can just use file_path.dir to get the parent path. And since this is the parent path, I can just say, I can rename this to parent path just so it's a lot clearer, right? And if we keep going down, we have the same thing over here again. Let's rename this to parent path and remove file_path.base over here, and just use file_path.dir. And that should be pretty much all we need to do for fixing the subfolders issue. So if I run this whole thing again, and keep in mind I've deleted everything in the programming folder, so we're starting fresh over here. If I run this again, it should work. All right, so the upload finished. Let's see if we fix the issue. You can see there's no API folder over here, right? If I go to auth, you should see, yeah, there it is, API folder over here, and then hello.txt file over here. So we fixed the subfolders issue. Now we can upload all sorts of different subfolders with the same names, just different paths, right? And it should upload them perfectly. Now there's just one final thing we need to do to complete this whole thing, and that is to map our subfolders from the tracker to our local machine. Basically, where am I right now? You see that the subfolders map we declare over here is empty by default, right? And this is because when we actually start to upload the folders, right, when we actually start to create and upload the folders, that's when we actually fill this map up. But if the project is already uploaded on Google Drive and the tracker over here that we're using says that all of the files are already uploaded and everything, then we probably won't be able to access our subfolders, right? Because we won't get to this folder, this path. We won't get to this block over here where we actually create and assign the subfolders. So what we need to do is that we need to go over to our tracker and make sure that we map the remote subfolders that we've already created on Google Drive to our local subfolders map over here. So that, for instance, if you have the API folder and you've already uploaded it to Google Drive, you need to make sure that you get the ID of this folder from the tracker and then make sure to add that to a subfolders map over here, so that if we have a new file inside this API folder, then we should be able to access that subfolder on Google Drive and upload that new file in there, right? Assign that as the parent. So, a lot of complicated words just to say that we need to map the subfolders. So let's create a function to do that. I'll create it down here. Let's go. This will be another pointer receiver like everything else. I'll call this `mapSubfolders`, and it will take the current subfolders map, which I believe is just a map of strings. And let's add some logs over here as well. Let's say, "Mapping subfolders from Google Drive to local." Then let's loop through the entire nodes list, `t.nodes`, and if the node is a folder, that means it's a subfolder. Then you can say the current map and the key over here should be assigned to `v.id`. There we go. And lucky for us, maps in Go are kind of like pointer types. You don't need to actually have a reference to them. If I just modify the current variable over here directly, it's going to reflect in the subfolders map over here as well. Now let's actually use this as well in `sync.go` down here. When we check whether the project exists, if the project exists, then we walk the project tree. And once we're done with that, because this `walkProjectTree` function is going to build the `t.nodes` key over here, right, which we need. So right after we're done walking the project tree, we should run `tracker.mapSubfolders` and pass in the current subfolders as the argument over here. And if we do this now, everything should work perfectly. So I'm going to start the upload process again, and this time I haven't cleared anything, so they should just skip all of the files because they're already uploaded and I haven't changed anything. Let's try running this, and you can see the upload just finished, and all of the files seem to be unchanged over here. So it looks like I didn't have to do anything. If you look up at Google Drive over here, you can see that the programming folder still only has the Drive Syncer video project that we uploaded before. If I look inside over here, all of the files are still here, and yeah, everything is working perfectly. So with that, we are, I think we're pretty much done with the sync algorithm over here. I'm gonna end the video right now because we've basically built like 99% of the tracker already. I think we're finished with it. There might be a few small tweaks that we need to do. We'll look at those in the next video, but I think for now we're done with this. So like, comment, subscribe, and thank you for watching.