All right. Welcome to our discussion of MongoDB's aggregation framework. The aggregation framework is a set of analytics tools within MongoDB, that allow you to run various types of reports or analysis on documents in one or more MongoDB collections. The aggregation framework is based on the concept of a pipeline. The idea with an aggregation pipeline is that we take input from a MongoDB collection and pass the documents from that collection through one or more stages, each of which performs a different operation on its inputs. Now, each stage takes its input, whatever the stage before it, produced as output. And the inputs and outputs for all stages are documents, a stream of documents if you will. Now, if you're familiar with pipelines in a Linux shell, such as Bash, this is a very similar idea. Each stage has a specific job that it does. It's expecting a specific form of document and produces a specific output which is itself a stream of documents. At the end of the pipeline, we get access to the output much in the same way that we would by executing a fine query. And by that, I simply mean we get a stream of documents back that we can then do additional work with, whether it's creating a report of some kind, generating a website, or some other type of task. Now, let's dive in a little bit deeper and consider individual stages. So, an individual stage of an aggregation pipeline is a data processing unit. As I mentioned that stage takes a stream of input documents, one at a time, processes each document one at a time, and produces an output stream of documents. Again, one at a time. Each stage provides a set of knobs or tunables that we can control to parameterize the stage to perform whatever task we're interested in doing. So, a stage performs a generic task, a general purpose task of some kind and we parameterize the stage for the particular set of documents that we're working with and exactly what we would like that stage to do with those documents. These tunables typically take the form of operators that we can supply, that will modify fields, perform arithmetic operations, reshape documents, or do some sort of accumulation task, as well as a variety of other things. So finally, the last thing I'd like to mention about aggregation pipelines is that it's frequently the case that we'll want to include the same type of stage multiple times within a single pipeline. For example, we may want to perform an initial filter, so that we don't have to pass the entire collection into our pipeline. But then later on, following some additional processing, want to filter once again using a different set of criteria. So, to recap, pipelines work with a MongoDB collection. They're composed of stages, each of which does a different data processing task on its input and produces documents as output to be passed to the next stage. And finally, at the end of the pipeline, output is produced that we can then do something with in our application. In many cases it's necessary to include the same type of stage multiple times within an individual pipeline. In subsequent lessons, we will look in detail at the various pipeline stages, what they do, and how to prioritize them. We'll also look at a number of different examples of building aggregation pipelines.
