Evaluation

Loading...

IBM 提供のコース

Data Science Methodology

4126件の評価

IBM

Data Science Methodology

4126件の評価

Specialization IBM Data Science Professional Certificate における 9 の 3 コース

Despite the recent increase in computing power and access to data over the last couple of decades, our ability to use the data within the decision making process is either lost or not maximized at all too often, we don't have a solid understanding of the questions being asked and how to apply the data correctly to the problem at hand. This course has one purpose, and that is to share a methodology that can be used within data science, to ensure that the data used in problem solving is relevant and properly manipulated to address the question at hand. Accordingly, in this course, you will learn: - The major steps involved in tackling a data science problem. - The major steps involved in practicing data science, from forming a concrete business or research problem, to collecting and analyzing data, to building a model, and understanding the feedback after model deployment. - How data scientists think! LIMITED TIME OFFER: Subscription is only $39 USD per month for access to graded materials and a certificate.

レッスンから

From Understanding to Preparation and From Modeling to Evaluation

In this module, you will learn what it means to understand data, and prepare or clean data. You will also lean about the purpose of data modeling and some characteristics of the modeling process. Finally, through a lab session, you will learn how to complete the Data Understanding and the Data Preparation stages as well as the Modeling and the Model Evaluation stages pertaining to any data science problem.

Modeling - Concepts2:55

Modeling - Case Study3:56

コースの提供者

Alex Aklson
Ph.D., Data Scientist
IBM Developer Skills Network
Polong Lin
Data Scientist
IBM Developer Skills Network

Welcome to Data Science Methodology 101 From Modeling to Evaluation - Evaluation!

A model evaluation goes hand-in-hand with model building as such, the modeling and

evaluation stages are done iteratively.

Model evaluation is performed during model development and before the model is deployed.

Evaluation allows the quality of the model to be assessed but it's also an opportunity

to see if it meets the initial request.

Evaluation answers the question: Does the model used really answer the initial question

or does it need to be adjusted?

Model evaluation can have two main phases.

The first is the diagnostic measures phase, which is used to ensure the model is working

as intended.

If the model is a predictive model, a decision tree can be used to evaluate if the answer

the model can output, is aligned to the initial design.

It can be used to see where there are areas that require adjustments.

If the model is a descriptive model, one in which relationships are being assessed, then

a testing set with known outcomes can be applied, and the model can be refined as needed.

The second phase of evaluation that may be used is statistical significance testing.

This type of evaluation can be applied to the model to ensure that the data is being

properly handled and interpreted within the model.

This is designed to avoid unnecessary second guessing when the answer is revealed.

So now, let's go back to our case study so that we can apply the "Evaluation" component

within the data science methodology.

Let's look at one way to find the optimal model through a diagnostic measure based on

tuning one of the parameters in model building.

Specifically we'll see how to tune the relative cost of misclassifying yes and no outcomes.

As shown in this table, four models were built with four different relative misclassification

costs.

As we see, each value of this model-building parameter increases the true-positive rate,

or sensitivity, of the accuracy in predicting yes, at the expense of lower accuracy in predicting

no, that is, an increasing false-positive rate.

The question then becomes, which model is best based on tuning this parameter?

For budgetary reasons, the risk-reducing intervention could not be applied to most or all congestive

heart failure patients, many of whom would not have been readmitted anyway.

On the other hand, the intervention would not be as effective in improving patient care

as it should be, with not enough high-risk congestive heart failure patients targeted.

So, how do we determine which model was optimal?

As you can see on this slide, the optimal model is the one giving the maximum separation

between the blue ROC curve relative to the red base line.

We can see that model 3, with a relative misclassification cost of 4-to-1, is the best of the 4 models.

And just in case you were wondering, ROC stands for receiver operating characteristic curve,

which was first developed during World War II to detect enemy aircraft on radar.

It has since been used in many other fields as well.

Today it is commonly used in machine learning and data mining.

The ROC curve is a useful diagnostic tool in determining the optimal classification

model.

This curve quantifies how well a binary classification model performs, declassifying the yes and

no outcomes when some discrimination criterion is varied.

In this case, the criterion is a relative misclassification cost.

By plotting the true-positive rate against the false-positive rate for different values

of the relative misclassification cost, the ROC curve helped in selecting the optimal

model.

This ends the Evaluation section of this course.

Thanks for watching!

コース一覧で検討

サインアップは無料です。今すぐサインアップして、パーソナライズされたお勧め、更新、サービスを利用しましょう。

コーセラは、世界トップクラスの教育を世界中に配信しています。最高峰の大学や組織とのパートナーシップに基づくオンラインコースが提供されています。

© 2019 Coursera (株)すべての権利予約します。

App Storeからダウンロード

Google Playで取得