Sync code with Google Drive - Part 1 (Google Cloud Console, OAUTH)

Alright, is this thing on? I think it's on. Right. Okay, guys, welcome back to another video. You guys know we've been building the GoRedis database clone thingy for a very long time now. I've recorded like 20 videos just on that one particular project. And I think I'm ready to move on. I think I'm ready to start building another project. And as you can see here, I've already started without you guys. So this is going to be a, and I call this a DriveSyncer. What this is basically going to do is it's going to synchronize, upload all of my programming projects to Google Drive as like a second backup. So before any of you smart asses tell me to use Git and source control, I'm already doing that. But it's just sometimes I just forget to upload all of my local changes from my machine to GitHub and everything, right? And that's exactly what happened to me very recently, just a couple of weeks ago. What happened was my hard drive basically just completely got corrupted somehow. I don't know what happened. I'm not a hardware engineer or something, but it got corrupted and there was no way to recover all of my data. And a lot of the projects that I built, I had a ton of code in them that I hadn't actually backed up to source control. So it was like you make a change and you edit like 20 different files and you just forget to push that to GitHub or source control or whatever you're using, and it just gets lost, right? And that was not a fun experience. It was a terrible experience. And so, yeah, I got traumatized. And to heal that trauma, I'm going to build this DriveSyncer project that's going to automatically upload all of my programming projects to Google Drive. And of course, this is not going to be like you guys know, I don't like to build simple projects. I like to go above and beyond and build something complicated just for the sake of it. This is not going to be just a simple upload folder to Google Drive project, right? There is going to be a ton of different features over here. Specifically, the most important feature that we need to build first of all, I mean, not in this video, but the one we're going to build in future videos, is that we want to be able to upload only the source code and not things like the credentials, the environment files, right? The ENV files and stuff, not the node_modules folder, not the folder, and so on and so forth, right? Not the, like this venv folder over here that gives us the Python virtual environment, not these folders. Basically, we don't want to upload all of these other folders that we wouldn't actually want to back up, right? And when it comes to GitHub and source control, you can easily configure that by just using this gitignore file over here, right? You can just put a bunch of file names and folder names in here, and source control will ignore all of these files for you automatically. Nothing needed on your side. But when it comes to Google Drive, there is no such feature. Like you have this app, the Google Drive desktop app that can basically just select a bunch of folders to upload. Where is it? Yeah, there we go. Yeah, there we go. So you can only select a bunch of folders to upload. You cannot ignore any folders, right? So if I just added a folder over here, which would be my entire programming projects folder, it will just back up every single file in there, all of the Git files, all of the node_modules, all of everything else, right? And we don't want to do that, obviously. So we need to build our own custom synchronization project over here that's going to do all of those things for us. Now that all I'm going to say about this project for now. I hope I haven't bored you guys from all of the long intros and stuff. But basically, we've built or I've built a pretty decent project so far. There's a lot of stuff over here, and yeah, we're going to build all of this from scratch. I'm going to show you how I built all of this stuff, and yeah, we're going to have a good time. So first things first, let's just throw all of this away and start a new project. I'm going to go do that right now, and then I'll catch you guys. Oh, and by the way, before I actually create a new project, let me just show you guys how this current project that I've built actually works. So real quick, I'm just going to do a quick run of this project, and, oh my God, I'm not in the right folder. There we go. Let's run this. And yeah, it's going to start the upload process for this folder. It's going to try deleting the previous folder so we don't create duplicates on our Drive and so on and so forth. It's going to upload all the files. It's going to skip the files that we want to skip, like this venv, the token.json, and this Git folder over here, and it's going to just upload everything else that we want to upload. And if I look here in my Google Drive, in this programming folder, I have this project over here, and these are all the project files, the test folder that I have over here, all of the files inside it. Perfect, right? So this works. It uploads all the files I want to upload. It skips the files that I don't want to upload, and yeah, this is a very nice project that I've built over here. So now that I've shown you a quick preview of what I've built so far, and by the way, we are going to build a ton of other features as well, not just what I have right now. But yeah, we all gotta start somewhere, right? So let's open up a brand new project and start building this whole thing out. Okay, here we are in a brand new project. I've already created a Go module over here. I'm going to create a main.go file just to get started. ```go package main func main() ``` And yeah, that's all we need to do for now. And by the way, as you guys can see, I did change my color theme from the Nord palette to this other thing. I like it, and I'm using this Are You Dark theme, and it looks pretty cool. I like it, but let me know what you guys think as well. Now, the very first thing we need to do is to create a new project in the Google Developer Console and enable the Google Drive API in that project, and also download the OAuth credentials from that project. And we need to do all these things so that we can actually interact with the Google Drive API, right? So this is the Go Quickstart documentation on this, on the Google Dev Console docs. We're going to just follow all of this step by step and create our project and get the OAuth credentials. So first of all, enable the API. Just click here. It's going to take you to the Dev Console. And once you're here, you can select a project or create a new one. Let's do that as well. As you can see, I already have this DriveSyncer project, but you probably won't have that. So just click over here in the project picker, click on New Project, and then you can just create a new project over here. I'm actually going to do that just so you guys can follow along perfectly fine with me. Just give it a name. I'm going to call this DriveSyncer Video. There's no need to give it a location or organization over here. This is unnecessary. Just click Create. It's going to create the project over here, and then just click Select Project over here. And you should see up top over here in the project picker that you're using this new project. Once you're doing that, just click Next over here. Click Enable API, and it will enable the Google Drive API. Boom. All that's done. Now we can actually close this, go back over here, and the next step is to configure the OAuth consent screen. So let's go over to Branding over here. Click Get Started. And we just need to add a bunch of stuff over here. So the app name, I'll just, again, just copy the project name over here. You can obviously put whatever you want. I'm just going to call this DriveSinker Video. Use the support email. Just add your own personal email over here. That's what I'm going to do as well. The audience, you can select as external. Contact information. Again, just put your own email address over here. I'm going to do the same thing. Next, finish, agree to the terms and service, and continue, and create. Okay, now that that's all done, we go over to this Clients tab over here in the sidebar and click Create Client up here. This is going to be a desktop application. I'm going to just call this, yeah, let's just call this Desktop Client One. We don't really care about the name. It's going to give us all of this data. We can download this, by the way, guys. Make sure to keep all of this stuff very safe, all right? Like, this is your client secret, and you do not want to expose this to any other third-party person, right? Otherwise, they're going to be able to use this project and use the Google Drive API and just modify your files, right? So don't let that happen. Download the JSON and store it safely somewhere on your computer. In my case, I'm going to store it inside the project over here. We're gonna go Reveal in File Explorer. I'm going to open this up over here and just paste it over here, and I'm going to rename this long name just to client.json. Awesome. We can delete this file over here. It doesn't mean anything. There we go. Okay, so we have this client.json. Let me format it as well, and this is what it looks like. And by the way, I am going to blur out all of the client ID, secret, and everything else over here because we don't want you guys to see this. We need to keep this stuff private. So yeah, but this is what it looks like. Now that we have our client, we can actually get started on coding, all right? So if you look at the Google Drive documentation again, after you've created your client, we've already done this, then you just create a new Go project, download these modules, and then write the code to actually get the Google Drive API access and modify some files. So first of all, we've already created a module and everything, so let's grab these two modules from the Google servers. So open up a terminal over here and just copy-paste these commands from the documentation over here. All right, so that's the first module. Now let's get the second. Why did I open up VS OBS Studio? Open up the browser and grab the second module over here. There we go. Boom. Okay, so we have all of the dependencies, all of the modules that we need to get started coding. So let's start writing some code. So first of all, let's create a get client function over here. And by the way, I'm just copying all of the stuff that they have over here in the documentation. You can do the same thing, or you can just code alongside me as I'm going to explain everything that we're doing as well. So a get client function is going to return a pointer to http.Client. It's going to take a config, which will be a pointer to an OAuth2.Config object, and it's also going to take a force refresh, which will be a boolean. And you'll see why this is important in, like, after we actually build all of this. But basically, we want to be able to force refresh the access tokens to the Google Drive API in case they expire or whatever, right? So if our tokens expire, we want to be able to force refresh them over here rather than just trying to use the old tokens constantly, right? So first of all, let's add the token file path over here, and in our case, this is just going to be client.json, right? This client.json that we downloaded from the Google Dev Console. And then we actually want to grab the token right from this file. So to do that, let's say token and error equals token from file, and passing the token file over here. Now we need to actually implement this function. So let's do that as well. This will just take a file, which will be a string, and it will return a pointer to an OAuth2.Token object or an error. So first of all, let's open up the file. Let's say os.Open. Pass in the file. If the error does not equal to nil, then return nil and error. And of course, use defer to make sure we close the file. And now let's just say token equals a new OAuth2.Token object, and then let's use JSON, the built-in JSON package, new encoder or decoder, actually new decoder. Pass in the file, and let's say decode and pass in the token over here. And then let's just return the token and return nil for the error. Now we have our access token over here. But what if there's an error in trying to get the token from the file, right? If we can't get the token from the file, then we need to get it from the web, right? From the Google OAuth screen where we grant access to the Google Drive API to use our account. So to do that, and by the way, this will all make a lot more sense once we actually run this and you see how it actually looks like and works. So for now, just understand if we can't get the token from the file, the access token, then we need to get it from the web. So let's implement that. Let's say token equals get token from web and pass in the config over here. And by the way, instead of having to declare two new token variables, what we should do is declare one variable over here, which will be a pointer to the OAuth2.Token pointer, and just use that in both places over here, right? As such. Now let's implement this get token from web function as well. Get token from web. This is going to take a config, which will be a pointer to an OAuth2.Config, and it will return an OAuth2.Token. No errors over here. So first of all, let's get the authorization URL. This is the URL that we need to go to to authorize the Google Drive API to use our account. So we'll say `authURL` equals `config.AuthCodeURL`. For state, we'll pass in the state token. Then we'll say `oauth2.AccessTypeOffline`. There we go. And then we can just say something like `fmt.Printf`: "Go to the following URL and grant the necessary permissions. Then paste the authorization code below." This is just standard OAuth authentication stuff. If you've worked with OAuth stuff before, then this will feel very familiar to you. And we can pass in the auth URL over here, and we can just say a new line, then `%s` to print the auth URL, and then another new line. And then let's actually grab the input from the user. So let's say `authCode`, which will be a string. And then let's say `if _ = fmt.Scan`, pass in the auth code over here. And if the error is not nil, if there was an error that occurred over here, then obviously we can't actually proceed with the application. So let's just run the `log.FatalF` and say, "Unable to read authorization code," print the actual error message after that as such, and then if this was successful, let's say `token` and `error` equals `config.Exchange`. We'll run this function. This will convert the authorization code that we get over here in this `authCode` variable into an OAuth token that we can use. All right, so let's do that. This takes a couple of arguments. First of all, the context. I'm just going to say `context.Background` over here, and then pass in the auth code over here. There we go. Handle the error as well. If there is an error, shut down the program and say, "Unable to read auth code," or actually, "Unable to retrieve token from web," and print the error message over here. And by the way, we should probably add a new line after both of these as well. Now, if everything went well, let's just return the token over here. Now, back up here in the `getClient` function, or actually, I should probably do a quick recap of this `getTokenFromWeb` function. In this function, all we're really doing is we're grabbing the authorization URL from this `config.AuthCodeURL` function. This is going to return a URL which we can use to grant permissions to that API. And then we just say, "Hey, go to the following URL, grant the permissions, then paste the authorization code," right? Then we read the authorization code from the `fmt` function, from the standard input. Then we just run this `config.Exchange` function, which will convert that authorization code into an OAuth token that we can use. And then we just return that OAuth token over here. All right, now back in the `getClient` function, we don't want to have to constantly grab the token from the web, right? Once we've grabbed the token from the web, we should try to save that token on our disk, right into a file. So let's run the `saveToken` function, which we will implement in just a second, passing the token file, `client.json`. And actually, this is a mistake. We don't want to use this `client.json` file over here. We want to add `token.json`. This will be a new file that will be created. So my bad over there. But let's pass in the token as well. And then let's implement this `saveToken` function. So let's say `saveToken`. This will take a path, which will be a string, and then `token`, which will be an `oauth2.Token`. There we go. And it doesn't need to return anything, obviously. Let's just say `log.Println`, "Saving credential file to," and then let's print the path over here. Then let's open up the file. Let's say `os.OpenFile`, pass in the path, and then pass in a bunch of flags. We'll say `os.Create`, so create the file if it doesn't exist, open it up in reading and writing both, and also pass in `os.Trunc`, which will truncate the file, basically empty the file every time we open it. So we don't have any past data inside that file, right? We want to completely have a brand new empty file every time we try to save the token. We don't want any old data over there. And then for the permissions, let's say `0600`. What this means is that the owner of the file, the person who created the file, will have the read and write permissions, and everyone else, no permissions. All right, so only the owner has the permission to read and write to this file. That's what we're saying when we say `0600` over here. Now let's handle the error as well. Let's say `log.FatalF`, "Unable to cache the OAuth2 token," print the error over here, and add a new line as well. And then make sure to close the file by using `defer`. And then let's use the built-in JSON package again. Let's say `newEncoder`, pass in the file, and run `Encode`, and then pass in the OAuth2 token over here. And that's all we need to do. Awesome. Now, back up in the `getClient` function, we should return the HTTP client over here, right? So let's say `config.Client`, pass in a context again. I'll just use `context.Background` for now, and pass in the token over here. And now we're done with this `getClient` function. Now we can actually start to write the main function. And by the way, you will see that this is exactly what the Google Docs say as well that we need to do, right? So let's actually, okay, so let's start implementing the, what is it, the main function. Let's say `file` and `error`. Let's say `os.ReadFile` over here, and this is going to return, it's going to take the token file, or not the token file, but the credentials file, which is `client.json`. And it's going to return a byte slice and an error. So instead of calling this `f`, I should probably call this `b`, right? And then let's say if error does not equal to nil, then `log.FatalF`, "Unable to read client secret file," print the error, add a new line, and there we go. Then let's run `google`. This is one of the packages that we downloaded. `google.ConfigFromJSON`, passing the raw bytes over here from the `os.ReadFile` method, and also pass in scopes. So if you're not familiar with OAuth credentials and OAuth authentication, scope is basically what permissions you want this client to have with whatever API you're trying to use, right? So in our case, we're trying to use the Google Drive API, right? So we can say something like `drive.DriveScope`. Again, this is also one of the packages that we downloaded. There's tons of different scopes over here, and you can see what each scope will allow you to do. The drive scope is basically the big daddy, I guess, of all of the scopes. It's going to allow you to see, edit, create, and delete all of your Google Drive files. You can also get the drive file scope, which will allow you to see, edit, create, and delete only the specific Google Drive files that you create with this app. And you can get all of the other scopes as well, like the metadata scope, which will allow you to see information about your Google Drive files, the metadata files over here, which will also allow you to manage and modify the metadata files in your Google Drive and all that. So you can pick the specific scopes that you need, the specific permissions and privileges you need in this Google Drive account. For our cases, we're just going to use `drive.DriveScope`, which is just going to give you all of the privileges on this Google Drive, and for our purposes, this is fine. So this is going to return a bunch of stuff over here. It's going to return a config and an error, so let's handle both of them. Let's say `config` and `error` equals this, and then let's handle the error as well. Let's say `log.Fatalf("Unable to parse, I guess, the client secret file: %v\n", err)`. Add a new line, and there we go. Now we can actually use this config object to get the client. So this `getClient` function that we spent so much time implementing, we can pass in the config over here. And for the fourth refresh, I'm just going to say `true` for now. We're not even using this boolean right now in the code. We're going to implement that probably in the next video, but for now, let's just hard code it. Let's just pass in `true` over here because this is obviously not going to do anything because this argument is unused in the function. But yeah, let's grab the client over here and then let's say `service` and `error` equals `drive.NewService`. This will create the Google Drive service. Again, pass in the context. I'll just use `background` for now. Again, for the options, let's pass in `option.WithHTTPClient` and pass in our client over here. Handle the error as well. Let's say `log.Fatalf("Unable to retrieve drive client: %v\n", err)`. Print the error, add a new line. There we go. And now we have the Google Drive service. We can now use this to create whatever files we want. Like, we can just say `service.Files.Create` and pass in a file, but we're not going to do this in the current video. For now, let's just run the list function over here, `service.Files.List`, and this will basically just give you the list of files in our Google Drive. So let's also limit that to the first 10 files. Let's say, and then let's run the `Do` function, which will actually run this API request and give us the result. So I will say `r` for response and `error` equals this function call. Handle the error as well. Let's say `log.Fatalf("Unable to fetch files from drive: %v\n", err)`. Print the error, add a new line. There we go. And if it was all successful, then the `r.Files` key will have the list of files over here. It will be a list of Drive files. So let's loop over this as such and just print the `File.Name` over here. Now, let's try running this. If we coded everything perfectly, then this should work fine. So open up a terminal. Let's say `go run .`, and it should run now. It's going to tell me to go to this URL and grant the necessary permissions. So let's do that. There we go. I'll select my account over here. Oh, okay, we also need to do one more thing. Since this is a new application and we haven't actually published this or submitted this for verification, we aren't going to be able to actually connect to this application until we add ourselves as a test user, which is why we're getting this access denied error over here. So close this. Let's go back to the Google Dev Console over here. Go over to the client, or actually, not the client. I forget where I was supposed to add the test users. In Audience, I think. Yeah, in Audience. Okay, so go to the Audience tab, and down here in this Test Users section, click on Add Users, and then just add the email address of the Google account that you want to use with this app. Click Save. Come on, save. There we go. Awesome. And now, if I go over to this URL again, it should work. I'll select my account again. There we go. And it's gonna give me a warning because Google hasn't verified this app. We haven't submitted this for verification. We're just testing this. Since we're the developer, we can just click Continue, and grant access to this application. Now it's going to tell us, "Hey, this app can see, edit, create, and delete all of your Google Drive files." That's the Drive scope that we added over here. So let's grant these permissions. Click Continue, and it's going to redirect us to localhost. We don't really need to worry about that. But if you look over here in the URL, it's going to give you this `code` parameter, which is exactly what we need. So let's copy the entire thing right over here, and let's paste it over here. Okay, now it's going to save the credential file to `token.json` over here. You can see it created this new file, and this is what it looks like. It has an access token, a refresh token, an expiry, and this `expires_in`, which is the number of seconds after which this token will expire, which is 3600 seconds. This is, again, I'm not good with numbers, but this is probably an hour. And yeah, it saved the credential file. Then it listed all of the files in my Google Drive, right? I said only give me the first 10 files that you can find over here, page size 10, and it gave those to me. Awesome! So I'm going to stop over there because I've been recording for like 46 minutes. Oh my God! And even when I cut all of this down, even when I trim out all of the bad parts in this video, it's still going to be like a 25, 30, at least a 30-minute video, which is insane. So yeah, let's end it over here. We built the OAuth 2 authorization stuff, and we basically interacted with the Google Drive API. Just asked it to list out some files from our drive. In the next video, we're going to continue with this and implement all of the file uploads and stuff and all of the other fancy stuff that we're going to do. This video is getting way too long. But hey, if you watch this to the end, then respect to you for having the attention span to watch such a long video. Most people can't even do that. Trust me, they can't watch any video longer than five, ten minutes. But if you made it this far, then you are amazing, my friend. Yeah, let's end it over here. Thank you so much for watching. Like, comment, subscribe, share the video. We are building a new project over here. We're done with Redis databases. We're building something even better now. So this was part one. Follow along because many more parts are coming very soon. And yeah, thank you for watching, and I will see you in the next video, guys. See ya.