Syncing projects with Google Drive - Part 14 (Deleting ghost files)

Guys, welcome back to the DriveSinker project. In the very last video, we built the functionality to overwrite existing files. So if you have, so for example, if you've already uploaded your project to Google Drive and you've made some changes to some existing file, let's say I add some new code over here and I want to sync that with Google Drive. Previously, it would have created a duplicate file, right? But now it's going to overwrite the existing files with the new file content. And that is just amazing. now in today's video what i want to do is i want to be able to delete ghost files so i explained this in the previous video as well but let me just do that one more time ghost files basically are files that are deleted from your local project over here but not deleted from google drive if you've been following along with our project over here with our drive syncer project series the entire time you'll know that our current implementation is not designed to delete unused files right which means that if I delete some files from my local project let's say I don't want this progress bar over here anymore if I delete that from over here in my local project it's not going to get deleted from google drive it's just going to sit there as like a ghost file that just takes up empty space but it doesn't really do anything meaningful in the project right so we're going to fix that so let me explain a bit about how we're going to fix this issue this going to be like a two-step process the first thing we're going to do is we're going to collect all of the local paths that we go through in our upload folder function over here where we upload the entire project we're going to collect all of the local file and folder paths in this project and then we're going to compare them with the file pads on the google drive right on the google drive remote and we actually have the list of those file paths in this t dot nodes variable over here right we use this walk project tree function in our tracker struct in order to walk the entire project tree on google drive right so we have all of the remote pads already what we're going to do is we're going to collect all of the local pads as well and then we're going to compare both of them and see which files exist on our local machine which files exist on our remote machine and depending on whatever mismatches we encounter over there we're going to delete the ghost files from google drive so let's get started the first step is to actually collect the local pads in our upload folder function in sync.go so right above the filepad.walkdir function let's create a variable called local local pad and this can just be a list of strings because each pad is just going to be a string and then down here just where we decide whether to ignore the file or not right in this ignore block just below this we're going to say local paths equals append local paths and path so we're going to append the current path that we're uploading onto local paths and once we do this we're going to have a list of all of the files and folders that we're uploading inside this project now the next thing we can do is we can go over to tracker.go and actually build a the function to delete ghost files. So I'm going to go all the way down here and we going to write a function This is going to be another pointer receiver as always And I call this delete ghosts this will take a list of local paths we call it local paths and this will be a list of strings and this will return an error if there is one we have the list of local paths and this is just going to be a list of strings but we also need the list of remote paths that is also going to be a list of strings we have the t dot nodes um map over here but this is a map and this is a map of nodes right not a map of strings so we need to convert this into remote pads right and the way we can do that is we can declare a variable call it remote pads i can use the make function over here and i can say make me a list of strings with a length of zero but a capacity of the same length as t dot nodes and this is just a performance thing this is just so we pre-allocate all of space inside this list so we don't really suffer the performance the performance hits of having to reallocate more space in this array as i append more paths to the remote paths list right that's why i'm adding the capacity of t dot nodes over here because that is how many elements we're going to have in this remote paths list right and then we can just loop over the um nodes list we can say for k in t dot nodes and you can say remote paths dot append and append k over here because k or the key in the nodes list is going to be the string that is the path to the file right and just by doing that we have a full list of remote paths and this is this is a list of strings just like our local paths which means since both of these both of these are a list of strings we can actually compare them much more easily now. I'm actually also going to create a list of ghost paths and this is also going to be a list of strings and then we can loop over everything in the remote paths list by saying range remote paths. As you know when you loop over a list the first variable you get is going to be the index of the list. We don't really need that because we're just concerned with the actual path so I'm going to ignore the index over here by using an underscore for the variable name and I'm just going to use the remote variable over here which is going to be the actual pad right now we're looping over every single pad in the remote pads list I'm going to say var is ghost equals true so every remote pad is a ghost by default unless we can find a match for it in our local project right so we're going to loop over the loop over the list of local paths as well and if both paths match if the remote path is equal to the local path then we can say is ghost equals false because the paths match that means that a local file exists in this exact same path as the remote file that means it's not a ghost right and then outside of that inner loop we can say if is ghost let's append this path to the list of goats ghost paths right And that's all we need to do over here to collect the list of ghost paths. Just by doing all of this, you're going to have a list of file paths that are completely ghost files and that are completely safe to be deleted. So now that we have the list of ghost paths let actually delete all of these files from our project on Google Drive right I actually add a log over here as well I call it I don know just add a bunch of separators and say ghost paths By the way guys, I know we've disabled the progress bars in this project's output. We are going to enable them later on, but right now I just like to have all of the logs over here on my screen so they're just easier to see instead of looking at in some sort of log file, right? anyway we're going to loop over the entire ghost paths list now and then we want to find the node on google drive that matches this ghost path so we can say something like if node okay equals t dot nodes pass in the ghost path as the key and if this is okay if the if this ghost path exists on google drive which it should because the entire ghost paths list is literally just a bunch of remote paths right so this should always be true then what we're gonna do instead of deleting the file is we're going to move it to the trash right so i'm going to create a new folder new new variable over here this will be a reference to drive.file and we'll say trash uh why am i not getting intellisense over here trash still not getting intellisense did i upload the wrong project uh there we go finally okay trash is going to be true and then we can just add a log statement over here say deleting ghost file print the file name or path whatever i guess we'll print the file path over here and that will just be ghost make sure to add a new line after the logs as well and then let's update this file on google drive with the new trash equals true um attribute so you can say if underscore error equals t dot drive dot api dot files dot update pass in the file id what's the file id in this case uh node dot where is it node dot id and then the new new the new metadata of this of this file which is going to be trashed and then we'll say dot do and then of course make sure that there's no error over here as well if there is an error then we should return it but i'm also going to add a bunch of extra information over here by saying fmt.error f fail to trash ghost file and then print the error message over here with the error message over here and also let's also print the actual file path that we failed on and maybe we can grab a few hints from that as to why the error was caused in the first place and over here if we are unable to find a node in this list then we can just return another error over here and say fail to find ghost file and print the file path over here as well this should almost never happen this should literally never happen because the the ghost pads is just a list of remote pads right so we should never have a case where we can't find the ghost path in the t.nodes list and that's all we need to do finally let's return nil for the error if there is no error and everything goes smoothly now this function is complete let's actually use this and to use this we're just going to go to sync.go again and at the very end of the upload folder function we're going to say tracker.delete ghosts and pass in the local paths list over here and now that we're done with that let actually test this out i going to create a new file over here i going to call it ghost and i just going to fill it with a bunch of random text over here as such right and then let upload the entire project again onto google drive it going to skip most of the files it's going to overwrite the files that have been changed so for instance over here you can see that we updated the sync.go file just now and we also updated the tracker.go files with the new delete ghosts function that we just wrote right so it knows that these files have been updated and it's going to overwrite them with the new changes as for everything else it skipped all of the files that were completely unchanged and it also uploaded this new ghost.txt file to google drive let's see it uh where am i let's actually go see this file on google drive and here it is there we go so this was uploaded perfectly fine now if i now delete this file from my local machine right now it's gone from my local computer and now if i run the entire sync program again let's see what happens if we implemented all of this perfectly then it should just upload yeah there we go it deleted the ghost file ghost.txt and it should be gone from over here yeah see we don't have a ghost.txt file anymore over here because we're deleting the ghost packs as for everything else we didn't change anything so i just skipped all of those files and folders and with that our delete ghosts functionality is working perfectly so that's all i wanted to implement in this video uh i have no idea what i'm going to build in this project after this i think we're pretty much like 99 or 100 done with the new sync algorithm that i wanted to implement and this is just so much better than what we had before previously we were deleting the entire project folder and then re-uploading every single file and that's just very grossly inefficient right this new algorithm that we've written is going to check each files unique each each files changes on its own and overwrite existing files and delete the ghost files and everything it's a lot more complicated but it's also a lot more performant right which means for instance if you're uploading tens or hundreds of projects with this with this application it's going to finish a lot faster if like you have let's say 50 different projects to upload and you've already synced most of them you're just running another sync operation over here it's going to see okay i've uploaded like you know 40 40 45 projects already i'm going to skip all of them instead of re-uploading them and i'm just going to upload the two or three new projects that this guy has built and that is just so much more efficient than what we had before now the video is getting a bit too long so i'm going to end it over here as well i have no idea what i'm going to build after this like i have not planned anything for this project past this point so i'm going to work on that but um yeah this video is over thank you so much for watching i'm not good with outros guys so you're just gonna gonna have to deal with that like comment subscribe do all of those other things that most youtubers tell you to share the video post lots of comments all of those wonderful things and i will see you in the next video whatever that may be thank you so much for watching thank you so much for following along with the series so far i hope you're enjoying this as much as i am and um yeah bye