Now to do that, you're going to click the Run Job, and within here, the behind the scenes again. If you're really interested in the process of creating these pipelines with Java or Python, it actually kicks off and creates a cloud Dataflow Job for you, which is great. Again, Dataflow is typically for more data engineering. Dataprep is data analysts and data scientists. I love working with the web UOS especially since I've been working with doing those same statistics with SQL for so many years, but if you're interested, and I'll show you what that cloud Dataflow Job looks like behind the scenes once we kick this job off. So default, it says. I'm going to create a CSV in this random location here. Now, that's not good enough. Go and edit the publishing actions which is where you want the output of this job to run, and keep in mind it's not just going to run on that 10 megabyte sample, it's going to run on that full BigQuery table that you have inside of your dataset. We want to not create a new file. We want to click over the BigQuery. We want to create new table. We're working on a BigQuery, so of course, let's create a new table. BigQuery into our dataset which is e-commerce, again created before. Create a new table and we're going to call this, we don't want all_sessions. Let's just call it a apparel_ revenue. A couple of different options you have. You can create a new table every run. You can append data to this table. It all depends on whether or not your input data source is going to be something that's going to have records for all time or every time you run this is going to be a new day, so do you want to append that data? Do you want to delete all that data, but keep the structure? What I'm going do is just drop the table every run and recreate it. Since if our Google Analytics that data's stored. Storage is cheap within BigQuery, we're just going to recreate that which is fine but those options are there for you. Save those settings and lo and behold, let's go ahead and run that job. Couple things while it's running. You can see a preview of the recipes. If you had more than one dataset that you're bringing in, for example, if you're doing a couple different joins and you're doing recipes on each of those different datasets and you're bringing them all in, you can get a really pretty looking directed acyclic graph here. This is also another place where you can just monitor the jobs. On the jobs panel, you'll see my windows is a little bit small, that's why I have of the Jobs Navigation here, but it should be up here for you guys. You will see that we are in the process of transforming. What this does behind the scenes is actually kicks off a Cloud Dataflow Job and we can actually do, all of this is running. This is all your historical jobs that have run. We can view the Dataflow Job. Well, all that begins to populate. Again this is just bonus behind the scenes. So, this is Cloud Dataflow. So, the cool thing is, the Cloud Dataflow and therefore cloud Dataprep since is just the UI layer on top of it, will actually expand elastic layer autoscale depending upon the amount of data that's being piped into it. Autoscale meaning, it'll fire up more workers behind the scenes to process more, and more, and more of that data so you don't need to worry about actually hardcoding the number of workers that should be working. It'll process a lot of that and it'll do it a lot of it in parallel as well. You've got a lot of these transformations that are happening here on the left in this flow and you can see where it can, it'll try to do tasks in parallel and eventually store it as a resulting table. If you're interested in building these transformations again yourself using Python or Java, the data engineering course is for you. We can keep track of the time it takes for the job to complete. We will monitor that and while that's running, I'm going to show you a nifty trick. Under Flows all the way back on the homepage, one of the things that you can do, that's just a one time setup. If you want to actually set this to be recurring, so say run it once a day, once a week at a set schedule. On the Flows page, hover over the recipe that you just created the Flow. What you can do click on More. You actually export these steps in the Flow which is actually pretty cool if you want to share that with somebody else, that's super useful. You can actually add a schedule associated with them, much like if you're familiar with creating cron jobs in the past. You can say, at this time every week on Saturday morning at, actually I just want Saturday, Saturday morning at,who cares, 03:00 A.M. for the Americas time zone, I want you to run this job. Then when you come back into the jobs Saturday at 03:00 A.M. you'll be able to see that a scheduled job has automatically ran. If you want to set it and forget it and not worry about creating those transformations and running them continuously, you can just set one of those jobs here and just come back in and monitor the jobs that you've seen here. We will come back to this job when it's actually completed and we will take a look at some quick job statistics, as well as previewing the final table that's created inside of BigQuery to make sure the insights are exactly as we transform them. It looks like our job has completed, took about eight minutes to run. Let's take a look at some of the stats. I'm going to click on the Job ID and it should take me to the job itself and you get a whole dashboard of stats here. You can see the final amount of rows that are outputted was 676. Again, I don't really use this that much. It will give you stats and the actual statistics and look little box and how much was plot down here for each of these different values. That might be useful. I actually really enjoy just taking a look at those inside of BigQuery. Going back to our jobs, you can see the execution here was manual that'll change when it's scheduled for that schedule that we just set up. Let's make sure that the Dataprep Job has done its output into BigQuery. I'm switching over back to our BigQuery dataset. If you don't see it here, I'm just going to refresh my page and boom check it out. We have a new table in there. There is another easy and easier way. If you don't want to refresh the page you can actually click on your project and then click refresh there in case you don't want to lose any information. Clicking into that, let's take a look. We have table details 676. Okay, great. Previewing similar data. All right, we've got timeOnsite, we got transactions, page type, all the different products that should be just apparel, and great. We've got the e-commerce action label that we set up and that looks good. Now you can continue running your insights on just this subtable. This could be refreshed again at the cadence that you'd like, but this is pretty much just a another raw of reporting table that you can then power into a Data Studio Dashboard or some other visualization tool. But imagine having multiple flows like five different flows for each of the different folks that are curious about getting their subset of data. The sky is the limit in terms of just just keep having those Dataflow Jobs. Just Dataprep and Dataflow Jobs, output those tables here into BigQuery and the dataset of your choice. Maybe you have a separate dataset that says e-commerce reporting. Dump those results here and let your stakeholders do with it what they will. All right, that's a wrap for the Cloud Dataprep lab. We'll go back and finish up the rest of the course.
