Syncing projects with Google Drive - Part 11 (Mapping local to remote projects)

all right drive syncer quick recap in the previous video we basically just we fixed an issue with our progress bar that was exiting prematurely before reaching 100 and more importantly we started building this tracker struct over here to track all of the file changes in our local folder on google drive and so on and so forth so you can actually only upload the files that were actually modified instead of uploading every single file regardless of whether it was modified or not and this is obviously not finished yet so we're gonna work on that now you'll notice that there's still plenty of problems with this tracker that we've built so far first of all the most important issue is that it's uploading duplicate files and the reason it's doing this is because we're not deleting the previously uploaded files over here and so we need to change that we need to start tracking with the files so that we can actually ignore the files that have already been uploaded and only upload the files that are modified right and of course make sure to delete their previous files so we don't end up with having duplicates and stuff so that's what we're going to work on now let me also zoom in so you guys can actually see and uh yeah let's get started so the very first thing we want to do is build a function called walk project directory so this will be another pointer receiver i guess let's say walk project tree and this function is going to walk through the entire project on google drive and basically kind of like make a whole tree like structure of the entire project on google drive right and it is going to be kind of a recursive function so might be a bit confusing to figure out but let's do this first of all let's just create a list of nodes a map of nodes it's going to be a map with string keys and node pointers as the values and these nodes are just the node struct that we defined in the previous video we're actually going to use these in this video and then we need a list of folders right to actually walk through so this is going to be like a queue. I'm just going to call it queue but it's going to be a list of folder IDs and the first folder that we need to walk through is obviously going to be our project folder. So I'll put the folder ID over here and we're going to start with this. We're also going to need to track the folder path of every single folder that we go through in this queue. So to do that let's create another list of strings or actually instead of a string list let's use a map over here that can hold strings there we go and then the first folder in this in this map is just going to be our project folder and the value will just be the path oh sorry the the key will be the folder id and the value will be the folder path you'll see why we need this in just a second now let's say for length of q is greater than zero. Basically, as long as there's a folder inside the queue, we're going to continue running this, right? Now let's say the current folder that we want to access is going to be the first element of the queue. And then the new queue is going to be queue and every element except that first element. So since we're processing it, we're also going to remove it from the queue, right? And then I also want to get the base path of the current folder that we're in, right? And And this is why we actually need this folder paths map so we can actually get the base path of this folder inside our project inside like our local project directory. Right now it's just going to have the current project folder right. But once we start to go through sub folders and stuff we going to need their base paths and that is why we need to have this folder patch map as well over here also let also handle um what it called pagination so let's create a page token this will be empty for now and google drive api will automatically fill this up for us we don't need to do anything over here but just in case there's like hundreds of files in your in your project folder you want to actually make sure that you're handling pagination here as well and just to just for now i want to have a bunch of print statements over here as well so i can actually tell what's going on i'll say walking the base path just so we know what's going on right and now let's create an infinite loop over here to actually get all of the files in this folder first of all we're going to need to write a query a search query so let's say fmt.sprintf now we want to make sure that the files we're searching for over here have the current folder right that we're processing in the queue right now have that current folder as their parent folder right and obviously make sure that they're not trashed right otherwise why are we even searching for them right now let's say request it's going to be where is my drive api there it is dot files dot list dot fields and then we'll say files and we're going to need a couple of fields over here first of all obviously the id the name the mind type but also the size modified time the sha-256 checksum and the list of parents we're going to need all of these and then let's also pass in our search query as well there we go and then let's also handle the pagination over here if the page token is not an empty string then we should add the page token to this request and then we're ready to actually send this request to google drive servers right let's handle the error as well i'm not going to do anything fancy over here for now let's just print the error and return if there is one right instead of returning we can also just break out of here and continue walking through the other project directories but yeah with that done Now let's access the list of files over here. I'll say 4f in range r.files and this will give us the list of files from the Google Drive API. Now if this is a folder and we can tell that from the mime type if the mime type matches our folder mime type then that means this is a folder on Google Drive right. And if that's the case then we need to obviously add this to the queue and make sure we walk this folder as well right so we'll add it to the folder paths by saying f.id over here and then we'll say file path dot join the base path along with this folder's name let's say and this will create the local path of this folder right local meaning the path on your machine right that's how you actually create like a tree of the entire project directory now to process this folder we also need to add it to our queue right so let's say queue dot append and add the folder id over here and this will make sure that we actually process this in the queue once we get to it right then once we're done with that stuff let's access the folder of this of this um file or folder whatever it is by creating a folder by creating a parent variable and this will create the this will hold the folder ID of the parent and then let's say if the length of f dot parents is equal to one then we can say the parent will be equal to f dot parents and the first element in there and we do it this way because there no guarantee if the file of the if the file or folder has a parent over here right and the reason i do it this way is to just make sure that there is actually a parent in that list right otherwise it could cause it could cause some sort of a runtime crash in our program and we obviously don't want that right now i also want to get the um local path of this file basically we're getting all of these files from google drive right but they also exist on our local machine right the project files and everything exist on our local machine as well so i want to create the local path of this of this file from google drive as well to do that just join the base path with this file's name now before we move any further we should finish creating this node struct over here because right now as you can see it's completely empty so let's finish building this as well because we're going to need a ton of fields over here so first of all the id this will be the file id on google drive and then the name as well the mime type which will be a string as well and then is folder this will be a boolean and we need to differentiate between files and folders on google drive and obviously we can use the mime type to do that but this is going to be a boolean over here also the file size this will be in bytes the modify time this will be a string the checksum the shot to 56 checksum and this is what we will use to verify the file integrity and finally the parent as well all right so that's all we need over here let's go back up over here in the walk function now let's say nodes and pass in the local absolute path and then create a node struct over here which will be a reference of course and then the id will just be f.id the name will be f.name the mime type will be f.mime type is folder this will just be uh where is it a conditional over here will say the mime type equals folder mime type if that's true it's a folder and it will be assigned true over here in this boolean field modified time f.modified time not modified by me time modified time checksum f dot shats with 56 checksum finally the parent which will just be parent over here now outside of this loop over here where we're looping over the files let's check if there is another page that we need to crawl and if there isn't if google drive does not return a page token for the next page that means there are no pages after this and we can safely break out of this loop otherwise if there is another page to crawl then we can say page token equals r dot next page token and then we can just continue running this loop until we run out of pages to crawl right and that's pretty much all we need to do to walk through the entire project tree right now outside over here outside all the loops but still inside the function let's say t dot nodes equals nodes this the nodes field on the tracker object and then just just for debugging just to make sure that we wrote all of that code correctly and there are no mistakes let's loop through the entire nodes list we'll say range t dot nodes and then let's print ln the key and then separate both of these with an equal to sign and then v dot let's say v dot id and then let's just print something like, I don't know, a couple of separators over here. And then let's say project three. And because really, I just want to make sure that this is all working right now. Since we built this we also need to integrate this To do that we go over to the sync file over here api slash sync and where is it down here where we check whether the project exists right if the project already exists then we need to walk the project directory so here i'll add a log statement over here as well i'll say um the folder name and then we'll say project already exists walking project tree and of course add a new line as well and then let's say tracker.walk project tree and then let's do a quick review of this function just make sure we didn't break anything we didn't add anything stupid let's add this statement up top over here as well so we know when it actually starts to you know walk the project tree and then let's run this program and see if it actually works see what happens okay so it looks like it walked three walk the entire project tree and now it's uploading all of the folders and files so while it does that let me just look at the project tree over here okay so these are two separate projects that we uploaded so let's go all the way to the top over here yeah this is a bit of a confusing output to look at because it's uploading two different projects at the same time and since this is happening concurrently it's just you know messing up all of my output right so let me go over to config and just sync only one project at a time so let's comment this one out let's remove this comma uh okay so apparently you also can't have comments in json so let's just remove it completely clear everything over here and then sync again okay so here we start it's walking through the entire project tree it's walking through all of these folders and here are the list of nodes along with their file ids on google drive and of course this also has the folders as well like sub like api and so on and so forth right so now we have every single file and folder in this in this project on Google Drive along with their file and folder IDs so we can actually access them directly on Google Drive and also along with their local pad I can literally use this local pad to open up this same file on my local machine right here so now we have a clear mapping between our local files and folders and the files and folders on Google Drive and you can add we can now use this to track which files have been uploaded which have not which have been modified which need to be re-uploaded which need to be deleted and so on and so forth and of course after scanning the project tree it's going to upload everything create a subfolders and all all of that stuff right now this video is getting a bit too long like most of my videos so I'm gonna stop over there and in the next video we're going to continue building this tracker functionality we built the functionality to walk the project tree so the next thing we need to do is to just add like some sort of a should upload function a function that tells us whether we should upload a file or not and then integrate that in our sync algorithm and then we will use that to decide whether to upload a file or not right and after that we should have the sync algorithm perfectly implemented so that's coming in the next video in this video we just wanted to build the walk project tree function and walk through the entire project tree and create a mapping between the local project and the project on Google Drive. Hope you learned something in this video. This project is coming along very nicely. If you did, leave a like, subscribe, comment, share, all of those wonderful things. And I'm going to see you in the next video. Bye-bye.