Moving the discussion with @amueller from pydata/patsy#77 (comment).
Proposing to:
- add an optional 'labels' argument to
log_loss
- add an argument to
make_scorer to enable passing labels argument to the custom loss function
Related to this - perhaps also allow make_scorer's need_threshold = True to be used for non-binary classification problems. Not sure why it's limited to binary. Totally meaningful for multi-class situations, where the metric might use the decision function to evaluate based on the rank order or something (for example, SVC doesn't have a cheap predict_proba, but has a cheap decision_function, so it would allow it to be used with rank-based metris). From make_scorer perspective, it just seems like a completely arbitrary restriction because it shouldn't care about what the custom loss function does with the decision_function output.
Moving the discussion with @amueller from pydata/patsy#77 (comment).
Proposing to:
log_lossmake_scorerto enable passing labels argument to the custom loss functionRelated to this - perhaps also allow
make_scorer'sneed_threshold = Trueto be used for non-binary classification problems. Not sure why it's limited to binary. Totally meaningful for multi-class situations, where the metric might use the decision function to evaluate based on the rank order or something (for example,SVCdoesn't have a cheappredict_proba, but has a cheapdecision_function, so it would allow it to be used with rank-based metris). Frommake_scorerperspective, it just seems like a completely arbitrary restriction because it shouldn't care about what the custom loss function does with thedecision_functionoutput.