Syncing projects with Google Drive - Part 14 (Deleting ghost files)

Guys, welcome back to the DriveSinker project. In the very last video, we built the functionality to overwrite existing files. So if you have, so for example, if you've already uploaded your project to Google Drive and you've made some changes to some existing file, let's say I add some new code over here and I want to sync that with Google Drive. Previously, it would have created a duplicate file, right? But now it's going to overwrite the existing files with the new file content. And that is just amazing. Now, in today's video, what I want to do is I want to be able to delete ghost files. So I explained this in the previous video as well, but let me just do that one more time. Ghost files basically are files that are deleted from your local project over here but not deleted from Google Drive. If you've been following along with our project over here with our DriveSyncer project series the entire time, you'll know that our current implementation is not designed to delete unused files, right? Which means that if I delete some files from my local project, let's say I don't want this progress bar over here anymore. If I delete that from over here in my local project, it's not going to get deleted from Google Drive. It's just going to sit there as like a ghost file that just takes up empty space, but it doesn't really do anything meaningful in the project, right? So we're going to fix that. So let me explain a bit about how we're going to fix this issue. This is going to be like a two-step process. The first thing we're going to do is we're going to collect all of the local paths that we go through in our upload folder function over here, where we upload the entire project. We're going to collect all of the local file and folder paths in this project, and then we're going to compare them with the file paths on the Google Drive, right? On the Google Drive remote. And we actually have the list of those file paths in this `t.nodes` variable over here, right? We use this walk project tree function in our tracker struct in order to walk the entire project tree on Google Drive, right? So we have all of the remote paths already. What we're going to do is we're going to collect all of the local paths as well, and then we're going to compare both of them and see which files exist on our local machine, which files exist on our remote machine, and depending on whatever mismatches we encounter over there, we're going to delete the ghost files from Google Drive. So let's get started. The first step is to actually collect the local paths in our upload folder function in `sync.go`. So right above the `filepath.WalkDir` function, let's create a variable called `localPaths`, and this can just be a list of strings because each path is just going to be a string. And then down here, just where we decide whether to ignore the file or not, right in this ignore block, just below this, we're going to say `localPaths = append(localPaths, path)`. So we're going to append the current path that we're uploading onto `localPaths`. And once we do this, we're going to have a list of all of the files and folders that we're uploading inside this project. Now, the next thing we can do is we can go over to `tracker.go` and actually build a function to delete ghost files. So I'm gonna go all the way down here, and we going to write a function. This is going to be another pointer receiver, as always, and I call this `deleteGhosts`. This will take a list of local paths. We call it `localPaths`, and this will be a list of strings, and this will return an error if there is one. We have the list of local paths, and this is just going to be a list of strings, but we also need the list of remote paths. That is also going to be a list of strings. We have the `t.nodes` map over here, but this is a map, and this is a map of nodes, right? Not a map of strings. So we need to convert this into remote paths, right? And the way we can do that is we can declare a variable, call it `remotePaths`. I can use the `make` function over here, and I can say, make me a list of strings with a length of zero but a capacity of the same length as `t.nodes`. And this is just a performance thing. This is just so we pre-allocate all of space inside this list, so we don't really suffer the performance, the performance hits of having to reallocate more space in this array as I append more paths to the remote paths list, right? That's why I'm adding the capacity of `t.nodes` over here, because that is how many elements we're going to have in this remote paths list, right? And then we can just loop over the nodes list. We can say `for k in t.nodes`, and you can say `remotePaths = append`, and append `k` over here, because `k`, or the key in the nodes list, is going to be the string that is the path to the file, right? And just by doing that, we have a full list of remote paths, and this is a list of strings just like our local paths, which means since both of these, both of these are a list of strings, we can actually compare them much more easily now. I'm actually also going to create a list of ghost paths, and this is also going to be a list of strings. And then we can loop over everything in the remote paths list by saying `range remotePaths`. As you know, when you loop over a list, the first variable you get is going to be the index of the list. We don't really need that because we're just concerned with the actual path, so I'm going to ignore the index over here by using an underscore for a variable name, and I'm just going to use the `remote` variable over here, which is going to be the actual path. Right now, we're looping over every single path in the remote paths list. I'm going to say `var isGhost = true`, so every remote path is a ghost by default unless we can find a match for it in our local project, right? So we're going to loop over the list of local paths as well, and if both paths match, if the remote path is equal to the local path, then we can say `isGhost = false`, because the paths match. That means that a local file exists in this exact same path as the remote file. That means it's not a ghost, right? And then outside of that inner loop, we can say, if `isGhost`, let's append this path to the list of ghost paths, right? And that's all we need to do over here to collect the list of ghost paths. Just by doing all of this, you're going to have a list of file paths that are completely ghost files and that are completely safe to be deleted. So now that we have the list of ghost paths, let's actually delete all of these files from our project on Google Drive, right? I actually add a log over here as well. I call it, I don't know, just add a bunch of separators and say "Ghost Paths." By the way, guys, I know we've disabled the progress bars in this project's output. We are going to enable them later on, but right now, I just like to have all of the logs over here on my screen, so they're just easier to see instead of looking at them in some sort of log file, right? Anyway, we're going to loop over the entire ghost paths list now, and then we want to find the node on Google Drive that matches this ghost path. So we can say something like, if node, okay, equals t.nodes, pass in the ghost path as the key. And if this is okay, if this ghost path exists on Google Drive, which it should because the entire ghost paths list is literally just a bunch of remote paths, right? So this should always be true. Then what we're going to do, instead of deleting the file, is we're going to move it to the trash, right? So I'm going to create a new folder, new new variable over here. This will be a reference to drive.file, and we'll say trash. Uh, why am I not getting IntelliSense over here? Trash. Still not getting IntelliSense. Did I upload the wrong project? Ah, there we go. Finally. Okay, trash is going to be true, and then we can just add a log statement over here. Say, "Deleting ghost file." Print the file name or path, whatever. I guess we'll print the file path over here, and that will just be ghost. Make sure to add a new line after the logs as well. And then let's update this file on Google Drive with the new trash equals true attribute. So you can say if underscore error equals t.drive.api.files.update, pass in the file ID. What's the file ID in this case? Node dot... Where is it? Node dot ID. And then the new new, the new metadata of this file, which is going to be trashed. And then we'll say dot do. And then, of course, make sure that there's no error over here as well. If there is an error, then we should return it, but I'm also going to add a bunch of extra information over here by saying fmt.errorf, "Fail to trash ghost file," and then print the error message over here with the error message over here, and also let's also print the actual file path that we failed on. And maybe we can grab a few hints from that as to why the error was caused in the first place. And over here, if we are unable to find a node in this list, then we can just return another error over here and say, "Fail to find ghost file," and print the file path over here as well. This should almost never happen. This should literally never happen because the ghost paths is just a list of remote paths, right? So we should never have a case where we can't find the ghost path in the t.nodes list. And that's all we need to do. Finally, let's return nil for the error if there is no error and everything goes smoothly. Now this function is complete. Let's actually use this, and to use this, we're just going to go to sync.go again, and at the very end of the upload folder function, we're going to say tracker.deleteGhosts and pass in the local paths list over here. And now that we're done with that, let's actually test this out. I'm going to create a new file over here. I'm going to call it ghost, and I just gonna fill it with a bunch of random text over here as such, right? And then let upload the entire project again onto Google Drive. It's going to skip most of the files. It's going to overwrite the files that have been changed. So for instance, over here, you can see that we updated the sync.go file just now, and we also updated the tracker.go files with the new delete ghosts function that we just wrote, right? So it knows that these files have been updated, and it's going to overwrite them with the new changes. As for everything else, it skipped all of the files that were completely unchanged, and it also uploaded this new ghost.txt file to Google Drive. Let's see it. Uh, where am I? Let's actually go see this file on Google Drive, and here it is. There we go. So this was uploaded perfectly fine. Now if I now delete this file from my local machine right now, it's gone from my local computer. And now if I run the entire sync program again, let's see what happens. If we implemented all of this perfectly, then it should just upload. Yeah, there we go. It deleted the ghost file, ghost.txt, and it should be gone from over here. Yeah, see? We don't have a ghost.txt file anymore over here because we're deleting the ghost paths. As for everything else, we didn't change anything, so I just skipped all of those files and folders. And with that, our delete ghosts functionality is working perfectly. So that's all I wanted to implement in this video. I have no idea what I'm going to build in this project after this. I think we're pretty much like 99% or 100% done with the new sync algorithm that I wanted to implement, and this is just so much better than what we had before. Previously, we were deleting the entire project folder and then re-uploading every single file, and that's just very grossly inefficient, right? This new algorithm that we've written is going to check each file's unique, each file's changes on its own and override existing files and delete the ghost files and everything. It's a lot more complicated, but it's also a lot more performant, right? Which means, for instance, if you're uploading tens or hundreds of projects with this application, it's going to finish a lot faster. If, like, you have let's say 50 different projects to upload and you've already synced most of them, you're just running another sync operation over here, it's going to see, okay, I've uploaded like, you know, 40, 45 projects already. I'm going to skip all of them instead of re-uploading them, and I'm just going to upload the two or three new projects that this guy has built. And that is just so much more efficient than what we had before. Now, the video is getting a bit too long, so I'm going to end it over here as well. I have no idea what I'm going to build after this. Like, I have not planned anything for this project past this point, so I'm going to work on that, but yeah, this video is over. Thank you so much for watching. I'm not good with outros, guys, so you're just gonna have to deal with that. Like, comment, subscribe, do all of those other things that most YouTubers tell you to, share the video, post lots of comments, all of those wonderful things, and I will see you in the next video, whatever that may be. Thank you so much for watching. Thank you so much for following along with the series so far. I hope you're enjoying this as much as I am, and, yeah, bye.