bvkwM.i-q[oPM;O
+UBj6l%]^""
bbbMPTTϾ߿!TBVAAAǓ&M1cAD"Hd!*?v\Ү@ pI!\ޠuaiwK^Bj I TEx:t()))W^t)a 5/Dd$qDDDEE}J$^Rr.iW,kZ'Hgg wIBPVs?=ϧiv)(rKB% ]CJ=W^o{yL$fNg_.8E =z4ѠAW^e_PTTcS!K ˲Yso
($Ib}Abm
zDK$Mq(jwIE qș4=^FEoM,_b(B($x gΛ7OTtGh!Զm;w(
a >ܩS'
mT2" L:kR2.zݻwʚ>}:BE3gDDٳg'%%4ݪU+Rz =լl}0*̇{`/iwv%S%C JeZ^*p^,d=xP(9n!$JJ%J$ ʸo6^n%e?hi=uNW d ./j9# \fQ^x/7䮹7H w7r:E˃mڱv n < phGCI8\ N ZVN|(*xwn" C
}w4H 9i펈(u_ 7ctvHa2h4dBPA ),VZ2ȿ/0J{[7` / gzOټ$ܻ iKc w 4ˏ{:]+^?gzDB3t A'])h (94:u˺M%H=@c
aquG'q+Had8 43;VCXPb_MA!} pc{:C\<DB[ o;lc 4p YW-,3B&ygSimq $xΞYN<@Fvv`d=;dsxl!dqBYqbfv3]H 307:06n@ 4PqxcX9#蘦 h T
9 8ASr3i!<< \?sإ}&5s /,))1ͮ
yG+T[ ߶#IhA@ 3ڬ&o \H.- R;v =id02-Hج7 a6)jWdi>^vN]G@nhwLŘ0CW=ab"N X`ӿ%Э`>Z;{4>,Ձ WBg*< |Kx0eyX?၆ɭyzy
0@@QcI88_0H$r>=!@소&oh2pT*eMWA] 1 CQA] 20Cn)4Nle|H3ۅ"ɞbMKmPU///oG\Zc㱊$I4
h M|^wvCWrU[Oe*DTr?^,zqBP$
TT*oW"!?Y%<86b᯿eqFdc2h4T*=+ӃN7db 7uu,ͭ/*lUdX7:,{A>0 -p0U{]\fg̈._G 5ԋȽq҂7u[Ocg8HݘŌ r-ܤF9<п?yI[8:[Y9]O G7ifc AF~e
-ah,{oD]p
i\"$7O(U;>~Ca>Z$p
U#>&=;}>h( 3>CQa6'H E1Q{;230 -p*H*خg e)aXBI ;pYtC"(3 5F31̘.f_RX\ʂ2 EvNݼ )cw"'[ ճr1ݻM6}}] |ТEG q 5 &-sU} WF5tAmjsrG gk1M̐ Ň+$ͳ<7Src5;P
L2>$aG
gx7{0;<< jJ 0` \Vxk͚5k ҲVHچy(A ևy4C$\m -[3-[^|9'PG%+.WyУ;+.,ĞSxq cxs\?RSSo٢Eh iYնm[y2JU
nC@1&THNw7sۋ6f]N:G g'vU0NJ\+n)k)Q^x`{M2}I⇰ ʎd~о`g'PdQ^K}v2pTgx[oi4O&r'Madk!hR3Wű,%l] jZ5A(оcֆ v2 䯿Сo%۷'m ,˳h˗Ϛ5kӦM 4X4Eas1_/cB }vX[X@. ";vǔv`A!0 -[$3g
5k8lٲ/gŊ6 Z EԴDuL gq 7ӧO <_ 6'a˰"R%Je_$ ziv$I DZ
Gq K(s
O_Qշ+j57|(xjvBלx>ȧ7Q!Z%q˦a2oCD&I۷;;ެ$ :,Y$""u_~g}vygrիWEEEߚD"-;6FW|>l}AD"mB} U\zX|O=bL#ʀyi)zDK` xSA`ְ`]k,
q g{g*jɒ%d@
f.SZJlbfд KxH 晶BXypv u'$$y$I^v&8 J4w_nFH F>+R%,omxڗ| /_|ׯ4={!CI ̕BO-]FF~*- R&uaxiɌ5Y` 7q6kދKNN/4Lew IGb̨PwP#y/ˤ} N;ɻsS) ȱlVի\|YN4h4nٲ壏>&8e-'ZJTt)2 L j ?OB6wll pJ_fMRR҈#vgϞUVm߾@CV{t!3*I¡+3dd{ +!D flc+ gϞGð|Nb
ZjJ d)ŧK ZFa^)0Ã
j3RO݀aȮ wgϞÇ#N<עs)uYwR51fTvQ$
7axjk<2D{vafoW
Vz/^zuddd;ubŊYfM6@"$^TJ }N'8~Pdj4˼\ ҃ a!ʿVA${ Kav(j$InڽyDhHq`0PEn:5mNhߌ,
/JJ[h_z%v3**s̙Oo
*'r)2 Z-ד$IQF1 (t}:BW)ŗ26|l *bP0FJ{c(j\ۺl$zd]PH ڐH$ [b4t Al ש!ӨEߙ=t}0jum搟i2nKVGLLpUL&+))qbhM'aX?oѳ.e0渐-OEDHCt!Wxiy-SQf6
xjڴ$ܳi˓$j jW5
8+;wP| :t@%''7oK(%|*U!qJ҄Z W3[_HPԳ.Ԅ!"'[3tS #_|144M6]rw裏M]Vl01t\5Twb2xd uֆ KʖwWu)L
:J qƞ={]Fo9bbÇ1[LS9A < *ͧoey O _۴sll )Ǎ`4x< BI6 K PL;fVNhq}5e)`]>
J/,,߿+Wq Q6m Lᎍ
z h4r h8^Ct]u@ <@^;O 7ĵ[966 g#8J6f}Za>⦏TzQ$; zan8/m 6hwBxip߯p=G[L& ê\ce= u5.7>lD 0x ʍ ڰdbŖ_3 <Wteʕ 6n11GkZ
K:d`
yKj W3BC|۴i@1H;7+t s-%j 19>>u]4ߞ<Z( BV _Uo5I%M#N<>wΝ;3 3tM6M>Ç
"88!ѣfͪ.[ZZ>ؚf%^wݵkW``I:veZtR4ܹ[&$$L0!7tP SRRV^6d0LR>` ĨT*4D+8$i0p}qHjM&]Hu_ӯU*IH-S93Tϲi6 KQS 5db1 ehuǛF z^okCXq÷ rgrz紽!;H$2Lz:G-0 ֭Ç_>q_|UVcƌYl0oݺu-X뛔tҼuY*qeee25ϫ7V5kF?xzzٳgϞ=fͪ}gfݺu .
b G ͞ 4h{Brի *LDÇlVD,F?l] D"M 03{ t:>;# gNQ@ 'Z ?c\u(I r )Ga|
R[_3r,04oD"! q~50СCsϞ=YS4֭[;vخ]_&;iҤ秤Im,PFXI={D =zo߾EFFG -]q;gٳgE =zEm~
"((rhOnURG
O囋Kؓc
pV$5/OdDX͛7o^B??J!ASh}uϳ^{^{}l9* !$5Q_$&MڵwϞ= F>ȑ#{$6P1I\(mo${Ua2Pq`{^/ܹ
//ӎP% xv?~x .p OOOwn\_tHh~b IDAT}}xK&(Z0fJvN>(\n|(ܽ /.ԎotF g'aÆY k4v fX, :thrr2Wb@i|$M(3IЁo P ܔo$&`N<7[;r)8 n}GA?~ÇkZV{رѣGs %w ίdl&:0\T\I__;͛͂ɌL wd Zo߾|>={^z" X*@6̫$U5Js= 4ڹ9y/{yN}<wwn0cT=YI >K233 ˯\yyMOj37DYqr@2Fp]tls'D۲,YRQQnT%Koz,-kG^478*y\Fg<+{ӭ^j @%Vn믻w &t>INbRF ʴͶb5<]/` L]4%+7@
!m 6?wJ=]WW0LC ~2Lgw^rr͛7 5j+je(9t
U~ n&yi7t }. nrAٓ !xddRV٩ 允M}i}oh~)S~S\qo?y~ko b%ܹsĉU*D.Y*3,p v7vs9-k\ 7 (KSr4ƈ0Ҵ g.BTvBss_%ں;yudEk /?Uv]+ ߟ?oQ/N BvdMvm7H)buegM^G& A E&rUܯO$YYyrׅʢ2_ i ɷu[ױ&/ mܗC&%?,\4|3 hѢ|^oP|Nf~(&D 8IMDp]^;ݚ((Zݠ!TfG .8hh{+$6PoU\(} {b `E% ٴ#^R,ܱ,jFl_1Og˟g˖-~~~gF} 9rCv89uԭ[FcfͺwhsP(9+f-=sټp2Г26U(;2$yy"!n^+y+X 3AƱ>v8qbpp0Bh˖-ʻvڼyso{ }3I&رlg}vΞ=[Bwy&U7YIrJxx8 =jӦ}}mAHs"
Am6T5,X`mZh6lҼ*!q^ѱד37HSPX=СB,ϼ400}Riqqq~~~HH[[eooo3;һHQԃBYYYcƌ={m8 uk _:{bfzþ
0Nch^l=k1MQLSf5 T)W/??}|E^myyyO>P*~~~BB_ S'q[gqcD'%% !lٲǏ;?0P?e7c))>dIAj]H`
wTc4-gL;jYK8d!Clْ_^^>uLDrŢ"Z?X^i4ԩ'NO>dA 8p <0LK. $_,XQVUaC=m8=ήҢE P~^zuݺuM.[O>aaa8/ZyQ5hР*wVf ;vÇwR(GHNN*Bw>A.k45RLvo[qqr|:=N9zxR9? ]8T*-))iP3ىDb'՟
{O9,9TT*$mP;b),±!3Y 3r&; ڞ>\ \lrr'6mT܆ ڵkFcTTTtttLLLDDN[f
7Ds^v!3,]Kjx 4mdIMu!61vtCF: >} ~h2"H$2O"-H$eR't]hƴTeEl#'.˘\wӽ{a^z5\.y>q V3*lb.8ê0]77暢niF X,u\nPHhϱ([! "
U*IF ~> .0Mw3_ q .|B~'5KX-rF?#Wx:Jֺ|[z5p*u|(jB/MEQF4}Th^@*a <:pT* TUU}
MEXlll &'
ZYhf_.aBPۢR(P?e% eE!2U2f c^ZE& p82ymN*C$㟉7{|4o[(x< AFnGY CBB\]]vxRRŋMM^vڕ Ǭmp1q7発B6eP4hyQgϞ={Go#}nfnNK
9RmEVv96x5-S5* r Hҹsy-Z b
t6T0Թ tåL 3|ʟJێkpy4' M'
1 ^QQQ`W~g///Cl]K5!GBZyB%m 888*'X#B* i: |BB
u5qmmm/_~%ia5p~(ݭ.\>p62s:q?48 2dȒ%K ~Hww~i&'''kkT^ZZiӦ ǃWX_G 6mZuz ݻyƌUUU >|_~9z]u 9sm{
O_|9::zϞ=O>=r䈹rĈ(}kwY O+u!.OZ6HleՖm",4BT*B?ZZ=Hk[@$M4ٳ ? 9rٳ G x7nܰׯ$nBBBqqΝ; qDFFJZv- Æ
?My'<|Ǐ?~ŋ76k988ԩS|I =/_ ng@TOyB%}j~vnTĮAuSVL9qekFHORO>V(vp^"x>:鶷s>ӳFիW;w ܵkW y>F7/ ߟ1L=O:::VTTxfee{Xllĉ5-[6yd`tݻwr$IÈ ֭[-Z4}^{_~ׯo$HkD-nN3o|u.ε RmۼcC"!8r -WNJ:8^AH2̓t#r~tk=tuuј]'_9 'HJJJkDWAf[
;vr8 E]]_QzKII\U[j rY;ĉ333CBBb%K̙)ovݺuT*=~-[ :(ݜ IZg|{uѵܹ
BBBQ]re]q\sEQwC ic@"舍 ?~Vӧ7mڴt
6,^Qbz666
,\n-aVVVxm_nq"<뀥*iތ%Iii)E"Y PXRRRU33sl;4ȱ{*d[`]}$Ieeemb1a lbK$2[,bҥݻw7E)hip KMM})t]
KK{#-(I8|o;o_M1 ixCz #/aY.8sn<ߣ&[n!zq3")G9F07ZMD6q EQECq_ rMOÎuiH #6qޠ:F?MA%&,rQp8f̀ ,Mb2>2[iGh4*q$]8 07SSU<
+5*Jz0c "CK @ -:
!kGQ˴zAXw i+ʴhJ8|oG j~DPDNVҺܾ_^ݚxr;;J&T( 9r`~ IDAT s :L8qҥ ݻw}YdddhNؾ}Ej B@LC̑Z=h#2iՁ@ pΝ] w gǎ ۵kW^?u HSyhnm Or%=gn2?mpp03?ox]͠ X,|}}3|ϟ{zz8+555++kk/]z;m'Gʔo|k)%e4?{4Y:cJ m.ZB袒zu۞r.99X3xiy&c;|ӧ8|`!-zPѥtOxAZ8<3 {
:PH2@ m5ɓ'Ϟ={MbcCBBFrMЅ :P(:u4w\ t
} 끢Io F8c-
0JVSpBQr٘^M ӳaO ;3Ȗl Y^4Mkڦqm"srƍ7n k۷oٷo߶m֬Y ߿? qinŵ|/ ?A
ߺ i"EULvp3/lQ b gC2نCBB>S \.GQT,:R'ɅkMh &\em.83+_li> R-3\zu6m >|̘1( 9rd֭ׯgzD_87z1[$i&$8j *uo;H 0/
nJnnuaW*M\.AMYRkF>IUPIߎ^{%%%88쨨+V<~8..β%+)
\d7ɍ5< OO6M p` ̲'pyח!df8 ଘ
e Ri'` |yK8ݽV-,= -)&xqZ4qa8x P}O uU:tիW *(*.\0gΜk5Q{5|pƌ7o UVV(JӴP( 9rݻ;w&''waĉK.ՅyNNNvwwt͋|BBu:._Tm)SO=j9 L@cKE R p& Z"j,! rsżxZ*" \JB &
̆խ[:v w^=bΝ;LZ ;v|ǟ}ٱcvիW'O|ᇉ ]ѣ~~~W^pvvnv KA+ 4dG3xtN*[o;@-f, {պ#J%B('P9b|Lh! > :ݽ{ pѣG 3gxzzΜ9!b̙۶m< {z{{;;;Ϙ1Ⱥu\\\||| :W7''gϞm۶ӯ:֭۩S.]PZZiӦ ǃWXk{e 9rժU) BHӡi*2uNX}|ft;Q] =4VlϐCSf,1 tsYwܻ#=]uw? âpɭ[ ٳC~L BBBΜ9SPP0iҤ~-55E >ϟ_xqr\y ?~x'NdddTkV{ٟ~ի7o<{^ݞ>}zʕaÆ u `SN5kuDDDPPбcǘ>7o1cƚ5kL|n MΝM!D7R>N.<0ŗ, ľzcfeeUYYi:7znsCLXK?O:8 ;7M`0:
qq-{?'op
z+[$WUurRzv L 2LP" ҥK}||(*uhM9)BSqU] 'wwY{/M % )e?HQC\ j\EWW@522=;"<Z"i{ LxͨnuuJnj& Szzz 8ɬ1! L (2 8By2ygX 1TJmIQy5cfHq ?P(d 0k!CJOwp2Sc٭Y !s-Twf@ &À裏"""ko)?S G4Ov VU+$ b&h֪{z!V 3wRR><< pʕfC_p J U q( k y94Ĥ~MʜiHx{@7fpC,DкC ֨ϙ3gٲe6mһWDVh2\xB<\
Y8
FSޅ8pER. h!iӥR7|æ62<>>4.&Y
+]ò3IwOŸ\1֖?:] 9𰴴 1bDǛX++$J/vД * L( q½?#UUًfI,;zp@X͛ qqq̆gϞ]tjAtUg?| h-J̱l,@Hu~i WVqNXWG A O^WNƍM%rM3EC,?\xI@"+$,= -)&xY WZ$ JCwb&4/_ 1!)G)e P=
wOK]C92/-b% ;1+oɎ !Irܹ~.Bbbݻ{51qRk!>"qG,xz;X Ni LI Rɽ7-PCt^.TR2pF'h!yaA.^kQ*7n/<<<֮]{ѣGQCva[\@S Nrh!2ukPA"Eժ
OW+tU~ k
иC "]\\"""o߾rnjS;on#ҕz8wVWk"e>0L ;`E?wpϜFu}q96VкC BU mbݮHf;22r ED(+@zsݬ_{B 2v.1ZX,6&baaa>ȳ'wWw j^e~n.;x= E9 M ho8SaRrrr(e2+7D4M;`ƍcU*#Au5 ((Z] Dj(-sEI9P6}]1 |>_Ւg_a~E.Uտ?AT*w_RJb쿂\.sjhixyQkzHs@Q3M% ٬Dj q6 U E.m. EC߶1`-[tw}{k gg縸6\Zƺ%gFtm#\WyqT%G
,3w VT@tQ NI-vxKQGv6; @ fĀizʕ
K
QQQ֭jݺu5j1 'ne}}혺 "iwEG3L
btU: `MAѷE.7+ĀݻwVV'+KLLL={ٳ':I';،뾸>)s,I0%mhro\A\a 1oEQtÆ
|MQ
#Se>|ޢM-#;fF2*su " ܹ;hРچ4lٲÇ_e5ئ (JuLHh -8c!pZV6nؼڢM( 2(3I4aN mNV&ioAJnCZ2 ?e] BdhIr h B2
傗N=F:ٱff_+ ZҶ>Ԓ$X6 ^*I*2#;I:Bhnu 0oٲeذsCݺu+ƌcz%??|> ?0Z
8 FFIiO%")ըr,@ ZjfAHЈJbC
RhhԨF
/J"-5$ُT<\u@ m
66$鴲4J5( (h&3'[cKx J$jQ)R4BӅ#D%BJUR T*s@Oc8iC E
|iCch Kcŏ[:eeKn]TČjukpi] ~71 akUUՄ HN5nnʕ*&
0@$4\M`րn v (vo{U*
ֽrR0#FA Ii^7|g('ؐc*X衑vןR2 'Ppŗ%
4nk@RjGquA1@`Jچ[0%(8%M`;wi@_4(8gA*&ɏR-;ٻra O=<8B͐q ۻwoVd vUP[[CjU@QpXZp:Z;HUa*k^ K~P+Ze@P ?riNWz'} _~(۸kVvm}W}Up
BګIeJj>/ٛ_m[qzecm ! !4BHZ: +:X\sܒw'.2 Y 춤
_ Up ;9-ml)&1sKZOGnm+(-V 0OE#`( k1Տ֩X ?=j澶ci6S] O!.: eN[9
t ` z/P;wи9 0 1R2G7nmwSSͮ|<_f ̓ڍ %X? f m ` < nWP(X < ! @x B 0 ` < ! @,] >|Фѣ!!!"صФ.77wʕص~~zbb ŋ>}aBss0EO!ǎ s玱T*=zhCC O}}ѣGe2 ۷ ?n*hUXXk*h[XXh*D! @'&[[[cBZ- GaZhp] MRT* 9r ӧOU* OKK
.L p h4
[4n111v-0 ӧW\g Tjb
-딖qy搿dddX0xˍ~itt ;>}jb
fee]GGKSE1Q^^G }xb\;XVV~իWdzBٹsgvv6EQMMM===۶maZ[[L:xoYG,md(ZT*JRVy/-kRZFGf~5/%%%t_{ߗnoo߷oߖ-[2228 Nfff7m{ˋ燆6hnn3g;ϟ7o nc퓓x<;mroLQB
%IDATer9< \^^4uTzիW@_6jKBBmq]4i!/'MۙuŊ۷oJ$WWWmWWWDBmdVO:5s!e+W :t---mhxyorȑYfy{{a݃` ?~LIJJjii^fې rSS<==ׯ_oRŻZ ,$13v7>;Н /**駟cccTC1HAAAo։'222ZdRt
=R
e-x燆쬬{u̓4ȖryMMg}& <:5A~+**VZ5tu 21cSjR__~BHCC'55ۺGI$'''BEQ|Ǐw1\vd˗/_rBp)))FdrQQ͛7cbb4Bo eww GYhQNNAE >}As`#TEl?{"{ｦ&ZyG&''ޖ111
Z6%%ܹsElڴIRiBg#gkrV?^ >4/+W$%%utttuu>}z]t62plɓ'qqq <9qĎ ;ld NLLloox0 %>>~2l9leeR&Oh"Bݻw.];رc4r@@@UU՚5kX,ք LvX_ly8d˳gϮ_n !wÆ
ٲP(OKK ?~III;
Ϧ;EQ .9uɓ'}]j9s!yyy۷oOHHzOLLܸqcqqV
J999'O/!===gϞΦ(?{/^5kV{{;!7ҥK| `2Nv=ƍ˗/{T~~͛7kkk=<<!gΜ;vlAAAkkkKKKYY
!D*=*jڵ"hƍsqq˫][PQl2]MMM ~*))),,Br1OOςŋBoܸ `n`&Laee^^^zԥ;!d"{o. !>}}}/_nii?IHHHHH $''?ϖR\\\zGjuYz//[n
)z0G|>ǰX,V7fjJ;fkeccSRR'v=z輼ݷo:4 nܸx}S&8tPE"QPPŋr.\
6~a߾}g͚oEGG;99
<8'''++kȐ!NNNu֮] 6dȐ/G8o
6xzz.[vss۷BxFFyyyrrr`` ܹw}gx-J1 x (yfPhlZo
"lǎ {n:.33ŧB :t/>S///$/\@о}w}%,+%%~ =vXqdd믿N?^d BhVz\k\\܋߿\"H}PZZڵkBqqq֭2e
rttOӧlvTTԪU>s/// 6%͛7c6bĈu},kԩz{{/_|۶m={Dedd'444)))77뾾gΜ)**JHHq6l@%&&~ ~a@@H$4 wBwIMM5 7"JO?!tAz?D 8p`tt ;wA u>y^/QPP4MXX&&x>@@?nN &!CFx}:;;T*-Za] x lgggJ:w\EEŜ9s {~-{#EEE7nHNNf =zqACa\|x֬Yj*RIovԩwcXbx
!@?vwwGxp+āN蓒.J͛7}^}vDDBfggA<<|Tz .l߾ݐ<==7m4ynݺvܹr)))ݻwo}yz
X,?O}nذa;{
W
?_|1rѣGggg=... ET*oH7|#o$%%eС111SNE_fϛ7dNNN}ܹsǎ ;jԨUV ?n`h-aaa|IaaÇϞ=w}}_w}IIIM ѻwoX <~ߺuĉΑ X,L&۴iS~ 69={ƅeee ,ׯH$ѣǚ5kt:fffN2Ϗƞ={R)}%K:vH{װCuu[o陘XRR2y ;v6t LPO5}0ܾ}W^\.w =v/Tzĉ(ý>>>۶m7n hE_.Zϯ}Vԩӓ'O͛w휜 , b!,,,77_ v_R%%%>>>L A 4 v< ` v< ` v< ` v< ` V]]]+** %Zv>>>JjY9SmxiM !y =zD!Cܻw߷o z) bvLP8q")) !T^^n(={ 9Ү]I&V<
:uQѺয়~:q;tj[~DndNNNɨ [gX.]JLLl]pp͵b
___PSS#vMo3u֊3gDѣG?x.Yz۲z}HHڵk}]T?,Xl'''zf;;;5oZ\|
߿Ϟ=mFĹs+++q ߱c{e2٠AjjjZ}v'''Xo߾VH t:SRR8 N Ʊ|hZ\9EQ)Uq[+lFC{*yȑ#
p Gu;55u RVx۷ &ܼyYas֭A$&&6Z vlvURjkkB!BHVUUDf3u֊ťA>>>NNNUBׯB-^hno$~ښ5kfΜܰ za}lܹ3?uyy "##[ܰ[븭Oeeeff&Y^^>vجԈCJUhh(UUUtVU( 7/^V . uyqyy9BÇT> $$7nڵɓ[V?[mx^{'O={vĈ!!!{G =w܈#tV[~w~WN0ۻ(jk* vy<^NN94͜9s}](2$22mxd2Yrrc||<߾}{С"kҤIu3gsqq3f[~.]PU `0j; hԩ7of:u<^? ؐz|=z`:u<^? gLEE+Q @ 4 v< ` v< `d.zNzjr/ ,xxԨQ,7oܸqT*1cA* @CZ
>==}ĸP"l߾}߾}>}:BHP\rܹ7n,))9|p
ЈO>>> 2 qPHB?u : .4VK t2N9s攖$''{zzbzXL/ BZee{G?2eʬYZ $9bĈx ۷jժ˗#0 wŧ 7!uVc*- 5pppdLGa@ (&BS*4B`:p rr@Lsrrj@L`X,KT2 ABTjͩ@:,ོHD _*2 innnR黐XAƜz{cALv +xJeY,?!D- Yi+Dl6VFaq"8 VtLbYlFͩVH/-K[wM6qH$8&zVNZ&VpJ%4ѷ l\.73$xA= ` hMw~@YuXh593 V(*]6əp & 4W_ӫɥJ !w9f3 kkk7oL?֭9ӣ ̩28)Pm"N HP1 cX 'X 6
ð\%eg\8
'&&Ɯ%m\9Xa'Sڪiوf+q0 EJl6ߞYbq
&qI&dgm`&V3ٵ:777Lɵ5B<[fN=0,\8 ń+!T!j~lH6G[>wNfʟ夣tYz%n3s8XM ޔBE!Y5yoW#s 69?% ]f+x3 yk߉K*"dkfC=0]E}[ Fն[s ^lퟬ[7|0 SWQYsoN21wq +T);) jx2 x\ ^B5+I
:85")p x@ շv[u!9"Te
9+ l xY`:-a2kw¯0~{=kި&]5%5*G Fԙr6s=x`>Eo]z>j ZS(LGaǣ(CsJ 4Zf:0 c*@TlJ%cJlDA~z5Vb9k}4Nݒ]}_'_7P(
]}m$M'Z)ꗗ]]ʏSg^+g)P5ʼn=%f;/z>00/8JqJ箚*E[*rFu/MRgW $1 kp8 Re)uppt *,J$o=qD魈O BVPA$2\dsNIw|U{Or"$K!j 0F1QD( ή< tZ42&ldu;Wc' ackk/}4`D︤3A 'w`^*Q t86IUƒ jeR ˚p߬%8k Qw`qSLc }WqtUԬ5ذ ۻf::`$x N .,\FT ;v,<< !uB{ݬB 釠ӽjL&,=!|I']< ?#3\!LZ% #gϞKIIh4tWz /!p7.)VMU*NꕸxD
!Tߛ n0{Z&z@bZ+#jjj4GGGl"T[9...*J&1 tGbxv^ ljܤR4ѷ|o;]jIG BLzgZ_;[%N lA f gn IFh7^ =
B^ٯuKtA X5L O# GV(KX4^{BzJ{)? q ;$d
wz ¤{pItYUZ,VeSX}w(YH +et'ي/zˤeMew;`:Fж tfyҽ' s浏l ߝ"áӽk7twtmHjnS 3 NwsX11[t
!Tݝ^T{f:F`9 ւ(*7uLcc$MWឱϚKRj1!0H w{|"t86ŸY>Y糛EC=w:$ 0i-^:[fyZr8{ZI
tA 0/ :yD;ñ)fy7*Y|q͕=N}0 < [ܣnI,z%.Iȟ *fyPӏ6 `qz=) N]TGC{(
ɒt^y:,ԧ ~tG-L4 H rށDto6
U䕟wi=ck
RUʤwodfc< rboF
ͥcDfg!\UpClF 2rt8H -S%F{
TVBRUQjr!!;'2^Z|fwHH\]]h
ߝ&ӽd!ǼG۩1YGaq5AOaS]+<)dcv~<o
+OnF:ݛBO*/9p<# -t Fg{0 tr/ L+ ͬ28q (&BSaMQ~"w>>U6Lq ʱTnm. !!D!Mڝwol831(_UDo}y8^g Bl6 3̮ks<,QAY8 G 0BSk&>),kH]Cah d'
y9@v bef һ^AR{It
$C(B֙ĆB\.g$BRg ZwF R.t{L
?Pm(d~P~0s.T۞VCl$x 0wp/Qg c:$@+W\/xD-K7),Pk
< E U#MD7W^5⑵,:{ U ,WL+ء .\Na Nw:9^Q|"'5{J2?6n04H !J%pF.0pl,( _P?Xƽd.%70"4 R'8 >''ĉW^H$z˫G C
4 ۜT)D4:bJ`H ;AFFƐ!C"##7lPSSӵkݻK>22r 0s7u}b4M +ޯUt?5b|?＃잫yrA&ns禥uWh4wO6mڴgϞ 0L K
V?M#'Kf5|* fMb~jTٯ9skH8qĉϟ?f QZݿӨӽnшG<,ʩ8z=>C(SA">$CP 8WUU;wÇ
aX/ Kdv&'YM#Be1 paF}N@;n
0.I! V>xL ݂6/kB$]*a9gPq͕66tS Kb lz~#oY>jg+xZ/|>oԩSϞ=m< `&'+ ̺%!}.;.Hg\5)ŮB#4L/\ ~0t{O2e/n -GmMt쌃$UMxbvQx }lbTf,(*C@L\ڵ믿8q"9a.;k֬UV}l (ųKgӽZTp]z,/U ~Lz{p{Nf"ƿUt+0DjAAA%~~~0
` Rq~c=oNw }~z_jLS0 'tlgݰݴRPQ0'D%@F )U 9OaH h ~?GGGT*m x1i
t7!Pu5lT^l7R9?7 ΦгNE n~r`- NJzHA fH f&u
۹sgYg8 5`=M oNaBBN] l*J3SW 0ន2!u=?xB;h" ^,߸q#--Ν;fycƌ!ɶX "žrs]{eܫn$BYۥ%Z?WwHTU=\zgH≌iO%+&:iehZ`L/YqN:ťmڴΝ;/ 5N"u9VZs !ÜeҲqָ=Qt.Kn =I 2)&Uϟw^//)SlY&-_2 +i"TS&QIԆ@irVA^|rssѝbAԨ l=Vk&k{!uL$իW$$$lٲ ;w.$x ,Irً%Koʶؒ4!){&Ez}npន8Íw`H 0_VV6x`бcdž aXvJJJ, 1J)1i%&O9??wVΡ!ta^Dn qmŽDر{ 8:s挷Eb Tdw(=8++JfsNc|xhP =222 j &.V iQ5A& mצ ԁVg69??????q8 hJ ÐϸjGLEՐRbgRZ&z 0{4(l!p#ݛ¿ ~Wa@v< ,α4+i" >_`xx-ƪ;`3\46W~GT*ed
-ÐIΎaH hmELj-2(I8T֊l} bEi4uBuJЯ;($9p8:N 0 cX)=* {|h^\FOYs _I쾛~6c?h4:mV7s5y/4?8Gpj^7 .!b6WwܩS㸇 f}?6e29/wpp`XfVb8DAPeZ)(GRƎՆwE!lF
` p <:%V*TeN]ϐ]Z7='jҰegq\Zm+ zbnAF5gPДbHT(}.z 8 R4 $xH2*b!lɓSRRedDFJĉl'Tī釈'5 ;x<@Hj N[C i~a:iݙz/p-*BT읂bgᰛuhO)Av۷o? =zm۶+Ww?a(sz=e2$qtL6CYLtkoR?n?K2oFuBgwfRY5Ia axZ`L\ׯW͠ GGٳg;w ~0 Q^I?DH4TR G +-zdg8!`$B-E-mx
CeHu&}6Bɩ
Ctks. 7uȦP2
!X9\{e ?C%/|{ߟjuo])f2 xIH}/v!Br/իBh;wD #0w4PG7"^̶P:.1ROVgvhZp zb^D3Tl0a"w
0͛'Nؿ'|rAD =Z3쫗t.ntzSH)!=Պת2Sq^ ,zH_C
TL$x__߬dffj4wy'!!f;88<~ cDA 7^S= bͤ"
{l5ϷXRUI;nH~ w3,
,LvAyzzjZOOO$Bbc`*%qzo_z7ɒ"Ib;םFR{Pd nDh`R7)3F ỳ sss###o zt{,
)=b@4[M준]KP~xc:mr} k`̜9slvNNΣG J H p\z]ѫӫ?{t_߬ a IDATdR C L$z 0 r (RB$ />$Ru%&K
Y& H C:4 ~ }ݻB }- ?opN E,R12ق1"V;`\u0kF+^ F"4R' aH &&5kbcc1 {qDDի- 6z8D<{"'lI%a Xz"n@bB #<]Ð:`]L$xww+W9s&;;ЁZB 0Og߸>wqWkwd: V-;%jE?Y4 6l2#'Zݮ X7BaϞ=B(**#eDI&j0
Zңs7yJRl >`AVHw}c s/r!A0X+` ItLd$]({̮wBSy8+GxOc$BcM~^N0XVp "0qZ|E0łcZN+ŋO Q5uu/e Z_j _>Dh CM?kFaZ
YK:WX0YED|Us<^`8,GF"40 RԽd;v ~wtZ%
6]u۶m|_|H$۷or! B
bʕ}Y@@_~y1c06xaJ%q:/ ٫]ݘU䕝u=ckqs{l'~ < 89%O/t% 0%hѢH1/H 6l d8 r : wiH2lPsBbX8le l"N )PrJDoHN01}apX,&[i~ݻwbܹs̙*++;w'p 4hЪUOd 33s{VUU
2ڵkϟ?~#GFj#Iܹs_kp OII7n>c @ݻwj;wݻ7B!4gΜҐdOOrbPL&۲e _+l1870VD62SJIkit\CW0gKf 0$OgS4rJu*p V<{-=~mL^u qlvw6j)jޣo gx{mߎ!us8 C@ sO<9w^zkԨQ8ܹð>(66ʕ+&kzPHQT-ZO?,[lҤIqqqt4nڴk={|<ŋn =i$I#Fq|߾}VZ|9+O (2CɓMbUUUSeD0ё(SJ K(^0FO
gRd2S4|>_*(: ^qd;u#wsXNI]]Ȑ6>nnn2LTlVv~r] k*+4F$I\^AkCsMMFSwvfqs3{ ѣG=<<Μ9sƍ zݻw;wljׯ B()))((!4}e˖mnnn7o&Ҍ:EQkMMM]t)SV //$@rrrBh"w;^ljkwX|bVNGI淫{;\d7!#]!DhH}$S3ۘ =zO4_~] 1B? ;;eh0S
NSN=vBh̙..-ozjnne<==z^G]pa
555JرcRY 4E/(N >H p
u$ ~oTqܟ NO.~~ ]H'u?%STZ? EG
˖-?r^3F~Q( M.]AAAiii6m\x1I;5++)m6` ! >{lϟ7o^JJF5 vJnz*I94l`
뼊*R܀yN~wfn^ >d*Fz>M~^0̙3W^?ѧO 7n?~!TPP۩S:})u66HV?{LyfLFwcԩSNSaɓ'O\0"׳^b_8C98MԵh tX AM61ROQ 3FU0Ӗ`ב >>ZG q8?~Ν;'O 4hPDDWXAQԇ ~صkЭ!ww˗K$u*qѣGUUUNNko^&իSRRlx̝;cڈNIREb}ԱKݱlŵWݙLq= =іcہِ:tmCMIIYj
=<<Ə d Î 92gwaQ\kϔݙ] f(&X5"XKbk z)c7ySLbł]1vET JYRי l{`#e|fO1hݥ"%_܊N] ܈J JX[|qƭ]O>>,XvZAWØ)_)2c:L
谐T
&Jsެ}C
7
;jXIIII999ߏ 9r_tJl@\AV%Do%x T\T\d"
~C}@Az
ݡ$ !ի?8qΖ/Z (zʕ8AOØ`I'BʎT f^yZIǺ(FjOGyj-kikԨQ;wtVdOx}A2yV/Z
P/N}G[G>!־o?sXXXZZNtǏ 0a !5o]T3m0ۗA=Yh(]A j燳0!fw}K հr?ܸqСC-_ |DEhHcGr< ϖX,Em܉s8qb0CBA e/RKXw :&;acM8xjCkUYI1ۺMvm;+ h #w;5п:+,e^8 ;tmy0 KII Z=a 9zpp\ ݿ* 5Ԉӭ|a\*72X' mhXhO9Bh4O)s+0j;$I>*GD3K>Ʃj+"^ҤH$j587}`&T*U# \B$+6_~9`>} \Bę3g[QeE
4q'Aq*TLK
00<w
1!%G0 S#,5huʃoʔuXۯlNkO\\ES߆r\J_hт O!wiha sUc䔧{qC*]&;#'T UT-+)J)!!l
A裏0rWhL\ Q@S z@u쏫N ;퐈E1JWJ3{
6JݻwAAAxxS G1 p%6"%i|#yGEzY%?Y˚h)iiz{hPMCri/ j |h)K@Duu} + >997 ?~|.]صC< uD ֍ *(D?~2lPf5*/0UcJvOԠsz7G> ux8e9ĶPO^9Zn҂ Ν/9xoQ^^JOHH uL&GEAְ! 0hg XRޡ%,U h>1a^o:twZW|UԬ"if'Rytvނ _Ν;Ϟ=[]]]XXpo rr{n+ &rȭ 5q0~9~H"#php!J"*`=1a^dA6]aZ3b_\Yر####**jӦMo#Gbq||Kaw]0 !cMQ 09,y~CUĄUgߝNޯ8"F$<3l,PVϋ fP ɾݔW֯y'vZsgϞΝ $$$DEE f͚pB. ʯDyyK/h~~~ = :$HrxfΜyex۶mS3f̰Lw#vj,?ԩ .t,';NO~*tB)/6 Op"U~gW P莝?? {-!B TTT4/\̙fBCC JrrO?dNxȑ#h!0SG5 ~ )9($'\ uGM a-P`R^q pX j HCRRRiYYm+ ɓwdw @HHڵkǏ oc ;v͚55[]ٱcG Qv5rz7Z^j,###m\·a-;qab{"[jq RA0 s\j 4Зu dn'?lϵ_8M dsI [Df{DC Oo/Qk6CV54.O>lٲDמnu r???c???z7Z(իW[ O:uٶGݠ8AB()C@Ov䀸Sfٍv4o1cgAwG©T&o畤jw 7q>^sJQxGavaC4/w?q8 /rzs| Ġjo? 0lذUV+111J >>5+r ލPD LzK?dl -I{38O,Wޛc9cKy1Ώ0Kdo?9>}/
!bw7g
^"\|̙3 ty[&Wo(000--R/ލPiQE5lL ^:g0 ңu ~1r+o8*@Εɓ'O(a%+VC;ξ筢PN r6xD #DZ!CGEE8%JO< OG0 װaK1&_ՊXNMцWWg"
23Upxoe!*l,@GD_~С۷oL#Gz7 9:53#b 1.AiTQ
U;=1a^c,M\}ȭ]^qr` *>)-`ud'k{+Jal
ܹ{BV5FM0Gvv~HmP& |b¼\s{)&嘽gwVRtrQ~jp} ǎg!ͳ fw% o=\ ?~aŋG CyG xb<O>wtW["N] TAQ*Tkx/<
` ee\ :ftՄ܍f:=` )9 P<~"L,e`Ϸ_~N ]),hu Pk"Hp8$I)St:S;v숏
k֬Ol۶o߾ ::z]vJ >| \A5++OJJcPBӆgwumYj YJQo]_h^ IDATP '&MgFE3,X!|x Lygddt:^K.}W >`Æ
7n,++ >˛
OԠ*T`wZ=0]YXB1̮A
*Fv 5k
F# aFӴV5kְaÖ.]χ9フ߷M6͝;7%%% `ԩh>3yDQ=ZRl`|&q4}%`J
gxNݺ]o:twZW|ۙ \I}I|0Bw rR~P;y, m@@A²ʕ+aÆM:588ԩSu.ɂjoxbbbb7k$͛7YfرP(ܷo߾}}! ~.y$WRA.6B1,- )9$|?EdA6gRb ]aA<:]ugy|K>P_PP`yT*U*T*Aeee=sK.-//H$FcJJƍccc333-O/JVfї < pرC"QZZjcC ĉ4ξTp0;?ĊxJ:qՙ]ׇP GM T>% tpݡV)==4M/[,11Ʋ"[n&MJJJ̤(cǎLJJ
6mڸq㲳irիgΜi \w)%%E"۷/--
p̙, EQX+㔗#{vy3, ΫW Ԏ0K"p7 @M ?6 욗]/~Nrl_`|A^F4$Pl 7@X;C@+p8^}
jժW^y%&&5 ;vܽ{ ~u/rڴi999aaaƍ[x56lذaëÐUVٱ 0L#O7d2ŋׯ_`ڵk+;\.qjKd =\ètT|>aW,.<"?-dbhTٲ4.K`& u+DO Cٍ;R]hp<ߋ$Iu:H$jK]
q T sC Je6a"H .vΝOm-;w+ Zrrr߿ ;rȋ/:f^QeE
caw .1g3.#&1$ OC0@~]q˫LMFJvGU'
w{ IIٶ-" (ʃN)M *B`ozaFdff&#s}YT^94 j]
ʯG'+ߞ4ﲣKN )`^P`\ }GCɓ'O(k0-Z~}ϚLqk
<y#^NIN XUg<]ҠC# cFN5i e3RLPT ӂJPV&!0|~~P(O
n8zeѿGNqin_U>aT[aO܈1筹R^̈?pY焧g%ť?V)Fx" ȅO] ֫W/GE
TFߋ#L1=] N XW^ V ejNrk\zRɺH#(>v,`\ cd( w{` N fDb4 wcVy!X|Z:XMw9roV^ᾡGnZ/+
fJR((I oȣ!=q sT~H"ayg3 Ғ HdZw^ Nұ LEK~W&x Jn&jimSO9CVRLOA:}XS(WTheiq?w=ڷ}1^l1=
23Upxo{І G(Jٲ$ݿR#=7ag\& | ƪSQ::S}DY@^HlEU`[E'vA|! =7hL7va0ݝ2<d
}J:XPvg @ǫXݭir? jMn6_#p8$Ive߾}fX,2ee&C ;v숏
k֬&l۶o߾ ::z]f, غuܹs܉rGM/%.Sac p Tʋ\
u0i٢mo]$S^O@s/
~)P^Bh7oԩV 'O,|> ###**
y5>?^zݻwoٙ\z}ǎ ;w>رc _˗g&Mp` Q*8~dE~ ~,Th P繦jj
"$-Qћ.=^wS2ldbf.7S
egԚ>ޟKYp \;'77G'R$&&yK?!!! Co>===::zׯ_ $$$,Y$55՞ wܹM {d~ڱc!(ǹ&a*ү0ݲ=y Q>.dn\vZgԚY%
rq!_~鈈:Oݿܹs+VhhS h^^^'N:VNUqP
NU'EsWh;&o9k, bCY3^ܯ<Ժ,,(>P61Ywk wr~lge~kϞ=Q5?P(l0iV;k֬aÆ5}ҥuvqР [Xe^`AdddttTDT 9
-)6K0> Ы#i *{Kq2 jglcɹwKuO
uNE&ӌ5~g?PӕCkQ^+
Ԕu:uzz+W,گ_>lذSӧNvLWVVV{{EEŋVJ2eJxxxvĉVС̙3 {siT&NBj)Uq+d}UƻQVɎ (+ʘi(Ե 9% #"$ '@T_Z_6හy,66vРAYYY{nҥ_kaFcJJƍccc333-O/L"ʒ[n駟>UV JJJviiyaMPfF2pK\C6ۇjJj <:s&oq.t\B$.vJ12̲*x^_HpH B$5ޚ4iRRRnݺ͝;722r[|ڴiƍ۾}{N]zYGcccPKJJ 2dҥK EQ:4MJJqtttΝm9:(zz )?#3^QeuzѲs,=-`D1fEhl?U攧 G)?YGDq[N#ͼG EZn{b=aX~J̒2>lұcݻw
mߺu_~9mڴq-^
6lذW_}aHHȪUF ݬ#w9|
y:f f͛3rrr>X\VV9sLJJJFay>^CQѣGQPݻwoVVg}Vr-x8WWW۲Je=B8ƺabĐcM(2 SoS4
Q!Q# `rg ~5v_6
-.tc`OJʶUTy;v!\.WV;(vVzbX30L$)
[I Y<=[Bqh"0_|L&kfL0rG~gAPy9^0hS'cfZQ"#GpշN..V^
=?:W17 `^Pc0
#rh/˛驉/k+Wg(Z(zܹ}-_f ;v,:y,<y#^NI 5C38IޱzQe[O5eDkb,F. ? :W^s̱,{۷lrڵk1;wVe@P7OXr6^+?c"oSSp8 d046p _| H4fsSg֫fn( 11077?}vHH &&СC۷OOO^~ K,IMM$we$Çǿˬ?)).Ƕ;jjez1 0;sLpר00x\ g+\=% A7=14@ÁԎtnq- y"ûE`OqߦiCEQRtsέXԩSdw //' =zTtْ-ٕ]];zhTj6bqUU/n:
(cW]qɤDpUwH4ޱzk,)Kga]jyYUvPauqZ7:O`$CJVK0 iZΚ5kذaK.}zzDDCCC>DԩK.egg ;PmRI%I PIh5ƻ kc][FZAI !fS ^Et Űrׇ
6uSNyL&
+++ŋO>Vb}F߿\E#""l?6d&6h֢mC38ՙ$XmB̴-rבY<4VWzpHFA Mc_ޗ Z
z.]Z^^.H,OƔ7fffZ_RSS- ]9#oݺuܹuJpܦB(3qvesNa#ϙT&(=T_0 f/k,1-" 7EwB7H$***z뭷&M[nyyysL\6mڸqoީSk^jhWvJ_|g}6i$)J9?0 j7 mB7\Id IDAT ] >%'k A7x9 vW)ʫ(
pTP @ytb ?aC[رݻ?í[~ӦM 7nŋ-ٰaÆ
^}Շ Z55=*n9]tζ0!my;q^4ϺA ?c)s.J#/P0}CeLH
6!.zqjvjU.و;t)ʮhuCU҆zJGQjWbZvg,biZWba"HPL(4___#(uK^FId<+!C8Oĉ4 Sd$wX*(AW6{ w}ƅ̍ 9Rɼv{Wx` \@A XI ,J:2V sE*+
{uu8 Ò+/qOdmE%H3?-] 1,: ANJvf%O2%<< C@{m:%Zn~kRb^QZoKci-#v'T4ށ6zh-_⋻{Tje:L JDΉB 'q Kj5UaPp!mIj,گ x E\ 2`@VvBy}hP{6&ؚN3NCQ̙37or-[233mۦVg̘m!jJeD%1wp<1O/C9NԜ)N?Rt\O8,[ب֖wT*8EC V{7vwy=A oE3`Or-C'KI|H0 A%_~e slWڧ ;v͚55[]i);jԨ]v$xVfy tf0 EQN =;RGYh8 j t #t( f~(͗ m"qĨo'6,COYzAAqV7L(J7=%N X
aGRˠ/vPekcp-+ q!ݺu ɐ~~~~~~+իW[ O:uٶ`fL#c֤7)iM{kM E. '"p+'
H -I=xcJ_'b+\:K XZJ_،xf=ӶG$N A3!< v
&oG
5}g R=jI,=(kKғ'OZ QQQaQp W(0 %qJg7ۢWTv߭i21 ;5X; Ǫ.sUl17Td (lɱ*.yTu*;!>NjI
z~Ju4U^ CWH0 Q H$2 _ă
h4X^0 V*6qq#V.Zh>k\ܺuX..o@%]@7^گqNK 50 ذ0C1j*/q9OK]-.KOﲽHТcSzD^,++1 I% _iSN%Bk0 O?ԙԫW^~mQQQ```ZZ+ Pf˯
2++ Ғ[[,2 Ivt7u%KZ,, K/ >|ӧ8DC_+WaY Nl6< ,+S]lިE!fq_-fb\fZw`I{p
Q~Swd ]\AS%~ xA[ b>ƾ~
%a^-PDA8ƥ](=K 6Չ̫F8֧3in*Lbjxk=0JOyq
+Bi$xa>裶0_ @@5# u<~ tSSR?W{&oI^gPB~N]1I_ A$ Yg3;l#m%3@[Rf<0lWy3lS E .1!+2VVVpv}aYZ+ &ս|oc#f5K(nf%D(=O[gl9+y`)KGBw(K+PtCDd.1yZfyҥKSSS MC͛wyǙGo}
Uل۲M0XmYf/5ٽF^9noć-"p| J%&bb$HT*]A}YI7nܘ1cF^^^JK/98JJJvI$=V ld!GE섲t/]y
AYI I7o;w5k(ZlFAQwvqPbց6"\n5ӗ/wAGlRymYO A:`Ǐ&%%1 hѢ={8'> L& ̙3,22r̙˧Ony1 bYML uqɲDž(@V
j)u-EI=YJպt .J)%p AqV<4M zy٤A-Y)b
6lԨQ(ݻwӦM})$_x : cg(j0 -ߕS#.V~ӗ8e0" `f p7:Ki xT(N ) fss` $)rٯ8q 8 EQfkk[gEqZzL&Kh1$}\2<<{+.\v&L\
1b5Oٳg|)m9q=ׁaXq>ƫA 3Y |zs
Jʎ4=v `4h_)u3AL&EQmZQ1 _R^uGf0$I^o;mge֭[
EJJJ~$k:'8s͟?_TcǎEGG;¬Frw8^>2ph `KvzAiz%V;ut5a9~S| 0_ ::s\= ^(dz@RUP(dhV,+-WS]W^@y7|ӣG:{ ΝsfZA^_iG
LuTd %f^g=F:{V賛~ 8.e
CO 2 LEZsgZ%ڈ%YfY'$.w n۷ύ } C\9[f2,VD~^p@ 6/
?SgF5Mu`/V;5 Qw =|+X^FiboXq ĉF# TE Eaq/ڻ^ R*]3䏨ԋKefjHW% An AqZiKFsN'T%-8|:V} >wg2%b9. e/2Q~>~>ZaYP35?S omKNi l=e1 l6ST";l{JI wVJ}8%VβMyX]V/'hZYR_! . #XI_~,Iʕ+A9s)ZqCL!*~G=ͭ4rKy6MypH ٹsgvv6aǏ8q"`0_ >l6;7]X)o9hza7=.p9n5& JT^EQo>+B5 0L&99Ν;oD}֭#G_oܸqϞ=ocIo pC>ZK7Vcޝ5u{ _H $IeUjUk QDVԩ>E:RTǡV9uhkQ4LۼŠ/%m8G(VT
nWoWx > 5Q*J?x>] @ 0 __B|^q @KmlM'&&;wZPsz{{_rXyyy qppMU*բE,SRRRSSmy
6M?%!l$;%~TPJ IѸf^'!"TaΣoMF{<RZV^g/~Nאá(mBpfs ӆ])mMcǎ=m|:u*99QnF1##j-oF<#X5\Y@$EX, ]OM+k*~w\Gl2 Z[[
rySS6 ÑJ6rJX|/HFPhϒ ?~ ;V*&$$dee9 TƯ"hroVA-^rΤs|gXLy~eu&(_ a=}
I&^o|YTT|)Sؿ?yyy}Ç>uTTTB1 uĨ5&9OzϔJ .|vvvrrr``hduuuӧO_~c+++[t)ׯ_zz\LpP$03t[s"4* eN e3lQ ~ĉΝvD"aX ,p8"ޔ4Wݳ4uĨ9Z5-D¬ Aj wkIII/F I1͖o1Xo(^2lV<:~N>|a[W Bw}7/)Eol ^xO ,`Μ|cMMB8 `O|Ϟ=G 5irw#b,Ҫf^ܽ1͡ I#NjE3;Yp8_lTٻJ#f
;|jZX1#wvHbx<*uZ|毺lt!eX@mOJ):w\nnnnnJJLL1cFBBw(uuuܽ冨Ö5k
-JG@ ]='
"5SƦa/
[N5ݒ].;Ő2D".l6[ 06!D*t:.r^F7ݓJt䱺r%Kߺu+!!aƌ'Off2u/egTfQkbaHq~0aN)b*Wo`zaaxX_g p]TQQw >|z̘1v1| ,
,8[a5! `///߿/_O/Zhڴi)ń9y]*Ey :RK=3K.MNNwpq6Wݧ k*gU m:}XѣG[N&B 8 :: EªGplޢ֗~/$p̏b :`BHyyyNNN' !sQe7 cuL΅ظ*
ЍZ k/;bwV%5QyE,DE+˥(dbF q\:
)q:y p
xaȜ|as*V7^!л !s
q0 w[X$1 @9f3MZv\¥t . |LYS]Sm4͑J
˹ Ǘ tA;CČw} @#s0dNުOL5 = sq| `'xb< x0dNTvBU:A$|?$0 )#0aNO oȰظ ! suF'w>[A O&(jO]:22dB x`OMkEkBq; 'Aӌ sl2 }b <6Lo~3xTO < NoCr,#k <VNG 0'eB>1 `>'߾O̮DB LhNי͛+oF Cp|[<}b sl-: p!?;M/qhտo߀ C[ۜ ЖNG x49?s' l'Co'ɟlҮPJ[Z'
sENҺV7a D- hSxީ>1O ؇GզuBj4=lD]U =yt;OZd'(p0! wSMڿ+TW[Z'B#q
# O#(U*v p<' OMڕ
U S!iSw*I>1 tx O 0& ̄D!T6gI`'y `ܲ漮ywgx+ 1)Ֆ>1z$Eή >1 BC p9 E%]|W7IJQ>1 Z<= Wf W/Q M; (X ׇ pCx 7 pCx 7T1}n B.hX -
!
!
!
!
!
(rv
LT*{9g>[>?i$g>FGG5م ټe˖~:&&ٵ]v%'':Ow_ >*ѣG8
/:
a6wQZZBG}} ;]
3? @g>j5͖d.}(
XBEQwuv-nh4*J\]C !L !< r7L/tp~mٲe?g}f2,;o>b] iqtjt1+nJk!}[ojժ:Zu
innnҽ.?P( _oZfD" ?}H rСC9
.\)Ͷ4%K Z/r̘1vh Rb=իuyڇΝ;|>ŊsYj~0g |]v%$$`A}Gw~z P(deAQTǏ 2dݻnݚajoll(;;53 -H
EQQG}` ~~~#G\e4Kܸq#!_6;;ʕ+{j\N9~8 ;vT*MHHr+a̙3.]JKK3L:.--m˖- iC2sL
r aT&DEEB 6ʼn'P<@JJVV*ܹc4m̶ۖZ6--l6gee 8pSNZl6o߾=;;ۑ/!h 67o\xg&4///##yk֬q av%%%w^j#_sh Sgw/^vmSSS߾}`2 LLL$ŕ-]bcWvq<#tܸq˖-#ļK+l RTxM6544CJ/--e >]goت
U n n nz~Сjd2Xm?̙3 Y 999Cm7UTT =zم #B؝`ؼymqi ` wٙ3gݻO>xw} cu\R#--MRYWUU%%%I'xСC"ի 'ܼy`MMͬYBBB222z=!dĉRT,7%IRRRyyYΟ?6B x [YYYeeesmii:ѣ ӝɓ'߸q믿gOyyI(2< !ȑ#+VXdVoeeeJJJ||ɓ'ӗ/_~Y xС={ |x>} 9RQQqIKnNwfΜ9c Kwܱ4nsܹӧO+JTJٹsgDDD~~)S!.\7 0 VBeuڵkt'_vMP <ؒ8bĈÇ$''?0`%K,Yy ~ ~X` l;b4f飢._# )z [yyyu{Ų:b6lh0 g;!
Ν;rH$ZvmXXؑ#G k'N;v n9mڴ>D" PVl !.88?߾}eUU ѿ\|YY={}X,;weF}ݺu\.}^7>C]zuРAIIIiii/^ ` ϖjۻ#FXnFBŋ)R*g
3gNMM7n/ţF*((rj1###|~hh諯)uԩSRe\ss3EQW
1cFyyyrrrbbb2ZҥK{%HNV^D7 7EQcǎ
0ۓJ3gMf詧jjj)ڵk۶myyy~^ 륗^jێkE͙3'''GT.[lܹOwBG };yx GxW
jug %
filter(Function == "auc", Package != "pROC") %>%
select(Package, Function, Description, Link)
rownames(auc.functions) <- auc.functions$Package
ci.search <- findFn("ci")
ci.functions <- ci.search %>%
filter(Function == "ci", Package != "pROC") %>%
select(Package, Function, Description, Link)
rownames(ci.functions) <- ci.functions$Package
# Get roc functions
roc.search <- findFn("roc")
roc.functions <- roc.search %>%
filter(Function == "roc", Package != "pROC") %>%
select(Package, Function, Description, Link)
rownames(roc.functions) <- roc.functions$Package
# Install missing packages
missing.packages <- auc.functions$Package[ ! auc.functions$Package %in% installed.packages()[,"Package"]]
if (length(missing.packages) > 0)
install.packages(missing.packages)
missing.packages <- roc.functions$Package[ ! roc.functions$Package %in% installed.packages()[,"Package"]]
if (length(missing.packages) > 0)
install.packages(missing.packages)
missing.packages <- ci.functions$Package[ ! ci.functions$Package %in% installed.packages()[,"Package"]]
if (length(missing.packages) > 0)
install.packages(missing.packages)
# Filter packages that are still missing
available.packages.with.auc <- auc.functions[auc.functions$Package %in% installed.packages()[,"Package"],]
available.packages.with.roc <- roc.functions[roc.functions$Package %in% installed.packages()[,"Package"],]
available.packages.with.ci <- ci.functions[ci.functions$Package %in% installed.packages()[,"Package"],]
#' Check if a function within a package is a generic function
#' @param pkg package name as a character string
#' @param fun function name as a character string
#' @return \code{TRUE} if the function is generic, \code{FALSE} otherwise.
#' If the package doesn't contain a function named `fun`, \code{NA} is returned.
is.function.in.package.generic <- function(pkg, fun) {
old.search.pos <- search()[2]
on.exit({
while (attr(parent.env(.GlobalEnv), "name") != old.search.pos) {
detach(unload = TRUE)
}
})
suppressPackageStartupMessages(library(pkg, character.only = TRUE))
# Does the package actually have a roc function
t <- try(get(fun), silent=TRUE)
if (methods::is(t, "try-error")) {
warning(sprintf("Package %s doesn't seem to contain function %s", pkg, fun))
return(NA)
}
if (utils::isS3stdGeneric(fun)) {
return(TRUE)
}
if (methods::isGeneric(fun)) {
return(TRUE)
}
return(FALSE)
}
# Test which packages have generic functions
generics.auc <- sapply(available.packages.with.auc$Package, is.function.in.package.generic, fun="auc")
generics.roc <- sapply(available.packages.with.roc$Package, is.function.in.package.generic, fun="roc")
generics.ci <- sapply(available.packages.with.ci$Package, is.function.in.package.generic, fun="ci")
# Prepare table
available.packages.with.auc$Generic <- c("TRUE"="Generic", "FALSE"="Not Generic")[as.character(generics.auc)]
available.packages.with.auc$auc <- sprintf('%s', available.packages.with.auc$Link, available.packages.with.auc$Generic)
available.packages.with.roc$Generic <- c("TRUE"="Generic", "FALSE"="Not Generic")[as.character(generics.roc)]
available.packages.with.roc$roc <- sprintf('%s', available.packages.with.roc$Link, available.packages.with.roc$Generic)
available.packages.with.ci$Generic <- c("TRUE"="Generic", "FALSE"="Not Generic")[as.character(generics.ci)]
available.packages.with.ci$ci <- sprintf('%s', available.packages.with.ci$Link, available.packages.with.ci$Generic)
# Final table
table <- data.frame(
Package = union(union(available.packages.with.roc$Package,
available.packages.with.auc$Package),
available.packages.with.ci$Package))
rownames(table) <- table$Package
table[available.packages.with.roc$Package, "roc"] <- available.packages.with.roc$roc
table[available.packages.with.auc$Package, "auc"] <- available.packages.with.auc$auc
table[available.packages.with.ci$Package, "ci"] <- available.packages.with.ci$ci
# Format as HTML table
htmlTable(table[order(table$Package), c("Package", "roc", "auc", "ci")], escape.html = FALSE, rnames=FALSE)
pROC/README.md 0000644 0001762 0000144 00000015746 15040443562 012354 0 ustar ligges users [](https://github.com/xrobin/pROC/actions?workflow=R-CMD-check)
[](https://github.com/xrobin/pROC/actions?workflow=test-coverage)
[](https://ci.appveyor.com/project/xrobin/pROC)
[](https://app.codecov.io/github/xrobin/pROC)
[](https://cran.r-project.org/package=pROC)
[](https://cran.r-project.org/package=pROC)
pROC
=============
An [R](https://www.r-project.org/) package to display and analyze ROC curves.
For more information, see:
1. Xavier Robin, Natacha Turck, Alexandre Hainard, *et al.* (2011) “pROC: an open-source package for R and S+ to analyze and compare ROC curves”. *BMC Bioinformatics*, **7**, 77. DOI: [10.1186/1471-2105-12-77](http://dx.doi.org/10.1186/1471-2105-12-77)
2. [The official web page](https://xrobin.github.io/pROC/)
3. [The CRAN page](https://cran.r-project.org/package=pROC)
4. [My blog](http://xavier.robin.name/tag/pROC/)
5. [The FAQ](https://github.com/xrobin/pROC/wiki/FAQ---Frequently-asked-questions)
Stable
-------
The latest stable version is best installed from the CRAN:
install.packages("pROC")
Getting started
-------
If you don't want to read the manual first, try the following:
### Loading
```R
library(pROC)
data(aSAH)
```
### Basic ROC / AUC analysis
```R
roc(aSAH$outcome, aSAH$s100b)
roc(outcome ~ s100b, aSAH)
```
### Smoothing
```R
roc(outcome ~ s100b, aSAH, smooth=TRUE)
```
### more options, CI and plotting
```R
roc1 <- roc(aSAH$outcome,
aSAH$s100b, percent=TRUE,
# arguments for auc
partial.auc=c(100, 90), partial.auc.correct=TRUE,
partial.auc.focus="sens",
# arguments for ci
ci=TRUE, boot.n=100, ci.alpha=0.9, stratified=FALSE,
# arguments for plot
plot=TRUE, auc.polygon=TRUE, max.auc.polygon=TRUE, grid=TRUE,
print.auc=TRUE, show.thres=TRUE)
# Add to an existing plot. Beware of 'percent' specification!
roc2 <- roc(aSAH$outcome, aSAH$wfns,
plot=TRUE, add=TRUE, percent=roc1$percent)
```
### Coordinates of the curve
```R
coords(roc1, "best", ret=c("threshold", "specificity", "1-npv"))
coords(roc2, "local maximas", ret=c("threshold", "sens", "spec", "ppv", "npv"))
```
### Confidence intervals
```R
# Of the AUC
ci(roc2)
# Of the curve
sens.ci <- ci.se(roc1, specificities=seq(0, 100, 5))
plot(sens.ci, type="shape", col="lightblue")
plot(sens.ci, type="bars")
# need to re-add roc2 over the shape
plot(roc2, add=TRUE)
# CI of thresholds
plot(ci.thresholds(roc2))
```
### Comparisons
```R
# Test on the whole AUC
roc.test(roc1, roc2, reuse.auc=FALSE)
# Test on a portion of the whole AUC
roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(100, 90),
partial.auc.focus="se", partial.auc.correct=TRUE)
# With modified bootstrap parameters
roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(100, 90),
partial.auc.correct=TRUE, boot.n=1000, boot.stratified=FALSE)
```
### Sample size
```R
# Two ROC curves
power.roc.test(roc1, roc2, reuse.auc=FALSE)
power.roc.test(roc1, roc2, power=0.9, reuse.auc=FALSE)
# One ROC curve
power.roc.test(auc=0.8, ncases=41, ncontrols=72)
power.roc.test(auc=0.8, power=0.9)
power.roc.test(auc=0.8, ncases=41, ncontrols=72, sig.level=0.01)
power.roc.test(ncases=41, ncontrols=72, power=0.9)
```
Getting Help
------------
* Type `?pROC` on the R command line
* Make sure you've [read the FAQ](https://github.com/xrobin/pROC/wiki/FAQ---Frequently-asked-questions)
* Search for [questions tagged with pROC-R-package on Stack Overflow](https://stackoverflow.com/questions/tagged/proc-r-package?tab=Votes)
If you still can't find an answer, you can:
* [Ask a question on Stack Overflow with the pROC-r-package tag](https://stackoverflow.com/questions/ask?tags=pROC-r-package)
* [Bug reports should be submitted to the GitHub issue tracker](https://github.com/xrobin/pROC/issues)
Development
-------
### Installing the development version
Download the source code from git, unzip it if necessary, and then type `R CMD INSTALL pROC`. Alternatively, you can use the [devtools](https://github.com/r-lib/devtools/wiki) package by [Hadley Wickham](https://hadley.nz) to automate the process (make sure you follow [the full instructions to get started](https://devtools.r-lib.org/)):
```R
if (! requireNamespace("devtools")) install.packages("devtools")
devtools::install_github("xrobin/pROC@develop")
```
### Check
To run all automated tests and R checks, including slow tests:
```
cd .. # Run from parent directory
VERSION=$(grep Version pROC/DESCRIPTION | sed "s/.\+ //")
R CMD build pROC
RUN_SLOW_TESTS=true R CMD check pROC_$VERSION.tar.gz
```
Or from an R command prompt with devtools:
```
devtools::check()
```
### Tests
To run automated tests only from an R command prompt:
```
run_slow_tests <- TRUE # Optional, include slow tests
devtools::test()
```
### vdiffr
The [vdiffr](https://github.com/r-lib/vdiffr) package is used for visual tests of plots.
To run all the test cases (incl. slow ones) from the command line:
```R
run_slow_tests <- TRUE
devtools::test() # Must run the new tests
testthat::snapshot_review()
```
To run the checks upon R CMD check, set environment variable `NOT_CRAN=1`:
```
NOT_CRAN=1 RUN_SLOW_TESTS=true R CMD check pROC_$VERSION.tar.gz
```
### AppVeyor Build Cache
By default, AppVeyor stores a build cache containing installed dependencies.
Sometimes you want to clean the cache, for instance if a
`Graphics API versionmismatch` occurs on R-devel, indicating that `ggplot2` was
built with a previous version of R. For this you need the AppVeyor
[API key](https://ci.appveyor.com/api-keys) to make a `DELETE` request:
```R
export APPVEYOR_TOKEN=""
curl -H "Authorization: Bearer $APPVEYOR_TOKEN" -H "Content-Type: application/json" -X DELETE https://ci.appveyor.com/api/projects/xrobin/pROC/buildcache
```
### Release steps
1. Update `Version` and `Date` in `DESCRIPTION`
1. Update version and date in `NEWS`
1. Get new version to release: `VERSION=$(grep Version pROC/DESCRIPTION | sed "s/.\+ //") && echo $VERSION`
1. Build & check package: `R CMD build pROC && R CMD check --as-cran pROC_$VERSION.tar.gz`
1. Check with slow tests: `NOT_CRAN=1 RUN_SLOW_TESTS=true R CMD check pROC_$VERSION.tar.gz`
1. Check with R-devel: `rhub::check_for_cran()`
1. Check reverse dependencies: `revdepcheck::revdep_check(num_workers=8, timeout = as.difftime(60, units = "mins"))`
1. Merge into master: `git checkout master && git merge develop`
1. Create a tag on master: `git tag v$VERSION && git push --tags`
1. [Submit to CRAN](https://cran.r-project.org/submit.html)
pROC/build/ 0000755 0001762 0000144 00000000000 15042620757 012164 5 ustar ligges users pROC/build/partial.rdb 0000644 0001762 0000144 00000000074 15042620757 014312 0 ustar ligges users b```b`a 00 FN ͚Z d@$ 12^`7 pROC/man/ 0000755 0001762 0000144 00000000000 15040455005 011626 5 ustar ligges users pROC/man/lines.roc.Rd 0000644 0001762 0000144 00000003732 13607143106 014021 0 ustar ligges users \encoding{UTF-8}
\name{lines.roc}
\alias{lines.roc}
\alias{lines.roc.roc}
\alias{lines.smooth.roc}
\alias{lines.roc.smooth.roc}
\alias{lines.roc.default}
\alias{lines.roc.formula}
\title{
Add a ROC line to a ROC plot
}
\description{
This convenience function adds a ROC line to a ROC curve.
}
\usage{
\S3method{lines}{roc}(x, ...)
\S3method{lines}{smooth.roc}(x, ...)
\S3method{lines.roc}{roc}(x, lwd=2, ...)
\S3method{lines.roc}{formula}(x, data, subset, na.action, ...)
\S3method{lines.roc}{default}(x, predictor, ...)
\S3method{lines.roc}{smooth.roc}(x, ...)
}
\arguments{
\item{x}{a roc object from the \link{roc} function (for plot.roc.roc),
a formula (for plot.roc.formula) or a response vector (for
plot.roc.default).
}
\item{predictor, data}{arguments for the \link{roc} function.}
\item{subset,na.action}{arguments for \code{\link{model.frame}}}
\item{lwd}{line width (see \code{\link{par}}).}
\item{\dots}{graphical parameters for \code{\link{lines}}, and
especially \code{type} (see \code{\link{plot.default}}) and
arguments for \code{\link{par}} such as \code{col} (color),
\code{lty} (line type) or line characteristics \code{lend},
\code{ljoin} and \code{lmitre}.
}
}
\value{
This function returns a list of class \dQuote{roc} invisibly. See \code{\link{roc}} for more details.
}
\seealso{
\code{\link{roc}}, \code{\link{plot.roc}}
}
\examples{
# Create a few ROC curves:
data(aSAH)
roc.s100b <- roc(aSAH$outcome, aSAH$s100b)
roc.wfns <- roc(aSAH$outcome, aSAH$wfns)
# We need a plot to be ready
plot(roc.s100b, type = "n") # but don't actually plot the curve
# Add the line
lines(roc.s100b, type="b", pch=21, col="blue", bg="grey")
# Add the line of an other ROC curve
lines(roc.wfns, type="o", pch=19, col="red")
# Without using 'lines':
rocobj <- plot.roc(aSAH$outcome, aSAH$s100b, type="b", pch=21, col="blue", bg="grey")
}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{aplot}
\keyword{hplot}
\keyword{roc}
pROC/man/ggroc.Rd 0000644 0001762 0000144 00000005703 14520415670 013231 0 ustar ligges users \encoding{UTF-8}
\name{ggroc.roc}
\alias{ggroc.roc}
\alias{ggroc.smooth.roc}
\alias{ggroc.list}
\alias{ggroc}
\title{
Plot a ROC curve with ggplot2
}
\description{
This function plots a ROC curve with ggplot2.
}
\usage{
\S3method{ggroc}{roc}(data, legacy.axes = FALSE, ...)
\S3method{ggroc}{smooth.roc}(data, legacy.axes = FALSE, ...)
\S3method{ggroc}{list}(data, aes = c("colour", "alpha", "linetype", "linewidth", "size", "group"),
legacy.axes = FALSE, ...)
}
\arguments{
\item{data}{a roc object from the \link{roc} function, or a list of roc objects.
}
\item{aes}{the name(s) of the aesthetics for \code{\link[ggplot2:geom_path]{geom_line}}
to map to the different ROC curves supplied.
}
\item{legacy.axes}{a logical indicating if the specificity axis (x
axis) must be plotted as as decreasing \dQuote{specificity}
(\code{FALSE}, the default) or increasing \dQuote{1 - specificity}
(\code{TRUE}) as in most legacy software.
}
\item{...}{additional aesthetics for \code{\link[ggplot2:geom_path]{geom_line}}
to set: \code{alpha}, \code{colour}, \code{linetype}, \code{linewidth}
(new in ggplot2 3.4.0), and \code{size} (before ggplot2 3.4.0).
The \code{group} aesthetic is always active since version 1.18.5 and is
kept for backwards compatibility.
}
}
\details{
This function initializes a ggplot object from a ROC curve (or multiple if a list is passed). It returns the ggplot with a line layer on it. You can print it directly or add your own layers and theme elements.
}
\seealso{
\code{\link{roc}}, \code{\link{plot.roc}}, \pkg{\link[ggplot2:ggplot2-package]{ggplot2}}
}
\examples{
# Create a basic roc object
data(aSAH)
rocobj <- roc(aSAH$outcome, aSAH$s100b)
rocobj2 <- roc(aSAH$outcome, aSAH$wfns)
if (require(ggplot2)) {
g <- ggroc(rocobj)
g
# with additional aesthetics:
ggroc(rocobj, alpha = 0.5, colour = "red", linetype = 2, size = 2)
# You can then your own theme, etc.
g + theme_minimal() + ggtitle("My ROC curve") +
geom_segment(aes(x = 1, xend = 0, y = 0, yend = 1), color="grey", linetype="dashed")
# And change axis labels to FPR/FPR
gl <- ggroc(rocobj, legacy.axes = TRUE)
gl
gl + xlab("FPR") + ylab("TPR") +
geom_segment(aes(x = 0, xend = 1, y = 0, yend = 1), color="darkgrey", linetype="dashed")
# Multiple curves:
g2 <- ggroc(list(s100b=rocobj, wfns=rocobj2, ndka=roc(aSAH$outcome, aSAH$ndka)))
g2
# This is equivalent to using roc.formula:
roc.list <- roc(outcome ~ s100b + ndka + wfns, data = aSAH)
g.list <- ggroc(roc.list)
g.list
# You can change the aesthetics as you normally would with ggplot2:
g.list + scale_colour_brewer(palette="RdGy")
g.list + scale_colour_manual(values = c("red", "blue", "black"))
# with additional aesthetics:
g3 <- ggroc(roc.list, linetype=2)
g3
g4 <- ggroc(roc.list, aes="linetype", color="red")
g4
# changing multiple aesthetics:
g5 <- ggroc(roc.list, aes=c("linetype", "color"))
g5
# OR faceting
g.list + facet_grid(.~name) + theme(legend.position="none")
}
}
pROC/man/are.paired.Rd 0000644 0001762 0000144 00000005733 13607143106 014142 0 ustar ligges users \encoding{UTF-8}
\name{are.paired}
\alias{are.paired}
\alias{are.paired.smooth.roc}
\alias{are.paired.auc}
\alias{are.paired.roc}
\title{
Are two ROC curves paired?
}
\description{
This function determines if two ROC curves can be paired.
}
\usage{
are.paired(...)
\S3method{are.paired}{auc}(roc1, roc2, ...)
\S3method{are.paired}{smooth.roc}(roc1, roc2, ...)
\S3method{are.paired}{roc}(roc1, roc2, return.paired.rocs=FALSE,
reuse.auc = TRUE, reuse.ci = FALSE, reuse.smooth=TRUE, ...)
}
\arguments{
\item{roc1, roc2}{the two ROC curves to compare. Either
\dQuote{\link{roc}}, \dQuote{\link{auc}} or
\dQuote{\link{smooth.roc}} objects (types can be mixed).
}
\item{return.paired.rocs}{if \code{TRUE} and the ROC curves can be
paired, the two paired ROC curves with \code{NA}s removed will be
returned.
}
\item{reuse.auc, reuse.ci, reuse.smooth}{
if \code{return.paired.rocs=TRUE}, determines if \code{\link{auc}},
\code{\link{ci}} and \code{\link[=smooth.roc]{smooth}} should be re-computed
(with the same parameters than the original ROC curves)
}
\item{\dots}{additionnal arguments for \code{are.paired.roc}. Ignored
in \code{are.paired.roc}
}
}
\details{
Two ROC curves are paired if they are built on two variables observed
on the same sample.
In practice, the paired status is granted if the \code{response} and \code{levels} vector
of both ROC curves are \link{identical}. If the \code{response}s are different, this can be
due to missing values differing between the curves. In this case, the
function will strip all \code{NA}s in both curves and check for
identity again.
It can raise false positives if the responses are identical but correspond
to different patients.
}
\value{
\code{TRUE} if \code{roc1} and \code{roc2} are paired, \code{FALSE}
otherwise.
In addition, if \code{TRUE} and \code{return.paired.rocs=TRUE}, the
following atributes are defined:
\item{roc1, roc2}{the two ROC curve with all \code{NA}s values removed
in both curves.
}
}
\seealso{
\code{\link{roc}}, \code{\link{roc.test}}
}
\examples{
data(aSAH)
aSAH.copy <- aSAH
# artificially insert NAs for demonstration purposes
aSAH.copy$outcome[42] <- NA
aSAH.copy$s100b[24] <- NA
aSAH.copy$ndka[1:10] <- NA
# Call roc() on the whole data
roc1 <- roc(aSAH.copy$outcome, aSAH.copy$s100b)
roc2 <- roc(aSAH.copy$outcome, aSAH.copy$ndka)
# are.paired can still find that the curves were paired
are.paired(roc1, roc2) # TRUE
# Removing the NAs manually before passing to roc() un-pairs the ROC curves
nas <- is.na(aSAH.copy$outcome) | is.na(aSAH.copy$ndka)
roc2b <- roc(aSAH.copy$outcome[!nas], aSAH.copy$ndka[!nas])
are.paired(roc1, roc2b) # FALSE
# Getting the two paired ROC curves with additional smoothing and ci options
roc2$ci <- ci(roc2)
paired <- are.paired(smooth(roc1), roc2, return.paired.rocs=TRUE, reuse.ci=TRUE)
paired.roc1 <- attr(paired, "roc1")
paired.roc2 <- attr(paired, "roc2")
}
\keyword{programming}
\keyword{logic}
\keyword{roc}
pROC/man/power.roc.test.Rd 0000644 0001762 0000144 00000026201 14114130125 015004 0 ustar ligges users \encoding{UTF-8}
\name{power.roc.test}
\alias{power.roc.test}
\alias{power.roc.test.numeric}
\alias{power.roc.test.roc}
\alias{power.roc.test.list}
\title{
Sample size and power computation for ROC curves
}
\description{
Computes sample size, power, significance level or minimum AUC for ROC curves.
}
\usage{
power.roc.test(...)
# One or Two ROC curves test with roc objects:
\S3method{power.roc.test}{roc}(roc1, roc2, sig.level = 0.05,
power = NULL, kappa = NULL, alternative = c("two.sided", "one.sided"),
reuse.auc=TRUE, method = c("delong", "bootstrap", "obuchowski"), ...)
# One ROC curve with a given AUC:
\S3method{power.roc.test}{numeric}(auc = NULL, ncontrols = NULL,
ncases = NULL, sig.level = 0.05, power = NULL, kappa = 1,
alternative = c("two.sided", "one.sided"), ...)
# Two ROC curves with the given parameters:
\S3method{power.roc.test}{list}(parslist, ncontrols = NULL,
ncases = NULL, sig.level = 0.05, power = NULL, kappa = 1,
alternative = c("two.sided", "one.sided"), ...)
}
\arguments{
\item{roc1, roc2}{one or two \dQuote{roc} object from the
\code{\link{roc}} function.
}
\item{auc}{
expected AUC.
}
\item{parslist}{
a \code{\link{list}} of parameters for the two ROC curves test with
Obuchowski variance when no empirical ROC curve is known:
\describe{
\item{A1}{binormal A parameter for ROC curve 1}
\item{B1}{binormal B parameter for ROC curve 1}
\item{A2}{binormal A parameter for ROC curve 2}
\item{B2}{binormal B parameter for ROC curve 2}
\item{rn}{correlation between the variables in control patients}
\item{ra}{correlation between the variables in case patients}
\item{delta}{the difference of AUC between the two ROC curves}
}
For a partial AUC, the following additional parameters must be set:
\describe{
\item{FPR11}{Upper bound of FPR (1 - specificity) of ROC curve 1}
\item{FPR12}{Lower bound of FPR (1 - specificity) of ROC curve 1}
\item{FPR21}{Upper bound of FPR (1 - specificity) of ROC curve 2}
\item{FPR22}{Lower bound of FPR (1 - specificity) of ROC curve 2}
}
}
\item{ncontrols, ncases}{
number of controls and case observations available.
}
\item{sig.level}{expected significance level (probability of type I
error).
}
\item{power}{expected power of the test (1 - probability of type II
error).
}
\item{kappa}{
expected balance between control and case observations. Must be
positive. Only for sample size determination, that is to determine
\code{ncontrols} and \code{ncases}.
}
\item{alternative}{whether a one or two-sided test is performed.}
\item{reuse.auc}{if \code{TRUE} (default) and the \dQuote{roc} objects
contain an \dQuote{auc} field, re-use these specifications for the
test. See the \emph{AUC specification} section for more details.
}
\item{method}{the method to compute \link[=var.roc]{variance} and
\link[=cov.roc]{covariance}, either \dQuote{delong},
\dQuote{bootstrap} or \dQuote{obuchowski}. The first letter is
sufficient. Only for Two ROC curves power calculation. See
\code{\link{var}} and \code{\link{cov}} documentations for more
details.
}
\item{\dots}{further arguments passed to or from other methods,
especially \code{\link{auc}} (with \code{reuse.auc=FALSE} or no AUC in
the ROC curve), \code{\link{cov}} and \code{\link{var}} (especially
arguments \code{method}, \code{boot.n} and \code{boot.stratified}).
Ignored (with a warning) with a \code{parslist}.
}
}
\section{One ROC curve power calculation}{
If one or no ROC curves are passed to \code{power.roc.test}, a one ROC
curve power calculation is performed. The function expects either
\code{power}, \code{sig.level} or \code{auc}, or both \code{ncontrols}
and \code{ncases} to be missing, so that the parameter is determined
from the others with the formula by Obuchowski \emph{et al.}, 2004 (formulas
2 and 3, p. 1123).
For the sample size, \code{ncases} is computed directly from formulas
2 and 3 and ncontrols is deduced with \code{kappa} (defaults to the
ratio of controls to cases).
AUC is optimized by \code{\link{uniroot}} while \code{sig.level}
and \code{power} are solved as quadratic equations.
\code{power.roc.test} can also be passed a \code{roc} object from the \code{\link{roc}}
function, but the empirical ROC will not be used, only the number of
patients and the AUC.
}
\section{Two paired ROC curves power calculation}{
If two ROC curves are passed to \code{power.roc.test}, the function
will compute either the required sample size (if \code{power} is supplied),
the significance level (if \code{sig.level=NULL} and \code{power} is
supplied) or the power of a test of a difference between to AUCs
according to the formula by Obuchowski and McClish, 1997
(formulas 2 and 3, p. 1530--1531). The null hypothesis is that the AUC
of \code{roc1} is the same than the AUC of \code{roc2}, with
\code{roc1} taken as the reference ROC curve.
For the sample size, \code{ncases} is computed directly from formula 2
and ncontrols is deduced with \code{kappa} (defaults to the
ratio of controls to cases in \code{roc1}).
\code{sig.level} and \code{power} are solved as quadratic equations.
The variance and covariance of the ROC curve are computed with the
\code{\link{var}} and \code{\link{cov}} functions. By default, DeLong
method using the algorithm by Sun and Xu (2014) is used for full
AUCs and the bootstrap for partial AUCs. It is
possible to force the use of Obuchowski's variance by specifying
\code{method="obuchowski"}.
Alternatively when no empirical ROC curve is known, or if only one is
available, a list can be passed to \code{power.roc.test}, with the
contents defined in the \dQuote{Arguments} section. The variance and
covariance are computed from Table 1 and Equation 4 and 5 of
Obuchowski and McClish (1997), p. 1530--1531.
Power calculation for unpaired ROC curves is not implemented.
}
\section{AUC specification}{
The comparison of the AUC of the ROC curves needs a specification of the
AUC. The specification is defined by:
\enumerate{
\item the \dQuote{auc} field in the \dQuote{\link{roc}} objects if
\code{reuse.auc} is set to \code{TRUE} (default)
\item passing the specification to \code{\link{auc}} with \dots
(arguments \code{partial.auc}, \code{partial.auc.correct} and
\code{partial.auc.focus}). In this case, you must ensure either that
the \code{\link{roc}} object do not contain an \code{auc} field (if
you called \code{\link{roc}} with \code{auc=FALSE}), or set
\code{reuse.auc=FALSE}.
}
If \code{reuse.auc=FALSE} the \code{\link{auc}} function will always
be called with \code{\dots} to determine the specification, even if
the \dQuote{\link{roc}} objects do contain an \code{auc} field.
As well if the \dQuote{\link{roc}} objects do not contain an \code{auc}
field, the \code{\link{auc}} function will always be called with
\code{\dots} to determine the specification.
Warning: if the roc object passed to roc.test contains an \code{auc}
field and \code{reuse.auc=TRUE}, \link{auc} is not called and
arguments such as \code{partial.auc} are silently ignored.
}
\value{
An object of class \code{power.htest} (such as that given by
\code{\link{power.t.test}}) with the supplied and computed values.
}
\section{Acknowledgements}{
The authors would like to thank Christophe Combescure and Anne-Sophie
Jannot for their help with the implementation of this section of the package.
}
\references{
Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson
(1988) ``Comparing the areas under two or more correlated receiver
operating characteristic curves: a nonparametric
approach''. \emph{Biometrics} \bold{44}, 837--845.
Nancy A. Obuchowski, Donna K. McClish (1997). ``Sample size
determination for diagnostic accurary studies involving binormal ROC
curve indices''. \emph{Statistics in Medicine}, \bold{16},
1529--1542. DOI: \doi{10.1002/(SICI)1097-0258(19970715)16:13<1529::AID-SIM565>3.0.CO;2-H}.
Nancy A. Obuchowski, Micharl L. Lieber, Frank H. Wians
Jr. (2004). ``ROC Curves in Clinical Chemistry: Uses, Misuses, and
Possible Solutions''. \emph{Clinical Chemistry}, \bold{50}, 1118--1125. DOI:
\doi{10.1373/clinchem.2004.031823}.
Xu Sun and Weichao Xu (2014) ``Fast Implementation of DeLongs Algorithm for Comparing
the Areas Under Correlated Receiver Operating Characteristic Curves''. \emph{IEEE Signal
Processing Letters}, \bold{21}, 1389--1393.
DOI: \doi{10.1109/LSP.2014.2337313}.
}
\seealso{
\code{\link{roc}}, \code{\link{roc.test}}
}
\examples{
data(aSAH)
#### One ROC curve ####
# Build a roc object:
rocobj <- roc(aSAH$outcome, aSAH$s100b)
# Determine power of one ROC curve:
power.roc.test(rocobj)
# Same as:
power.roc.test(ncases=41, ncontrols=72, auc=0.73, sig.level=0.05)
# sig.level=0.05 is implicit and can be omitted:
power.roc.test(ncases=41, ncontrols=72, auc=0.73)
# Determine ncases & ncontrols:
power.roc.test(auc=rocobj$auc, sig.level=0.05, power=0.95, kappa=1.7)
power.roc.test(auc=0.73, sig.level=0.05, power=0.95, kappa=1.7)
# Determine sig.level:
power.roc.test(ncases=41, ncontrols=72, auc=0.73, power=0.95, sig.level=NULL)
# Derermine detectable AUC:
power.roc.test(ncases=41, ncontrols=72, sig.level=0.05, power=0.95)
#### Two ROC curves ####
### Full AUC
roc1 <- roc(aSAH$outcome, aSAH$ndka)
roc2 <- roc(aSAH$outcome, aSAH$wfns)
## Sample size
# With DeLong variance (default)
power.roc.test(roc1, roc2, power=0.9)
# With Obuchowski variance
power.roc.test(roc1, roc2, power=0.9, method="obuchowski")
## Power test
# With DeLong variance (default)
power.roc.test(roc1, roc2)
# With Obuchowski variance
power.roc.test(roc1, roc2, method="obuchowski")
## Significance level
# With DeLong variance (default)
power.roc.test(roc1, roc2, power=0.9, sig.level=NULL)
# With Obuchowski variance
power.roc.test(roc1, roc2, power=0.9, sig.level=NULL, method="obuchowski")
### Partial AUC
roc3 <- roc(aSAH$outcome, aSAH$ndka, partial.auc=c(1, 0.9))
roc4 <- roc(aSAH$outcome, aSAH$wfns, partial.auc=c(1, 0.9))
## Sample size
# With bootstrap variance (default)
\dontrun{
power.roc.test(roc3, roc4, power=0.9)
}
# With Obuchowski variance
power.roc.test(roc3, roc4, power=0.9, method="obuchowski")
## Power test
# With bootstrap variance (default)
\dontrun{
power.roc.test(roc3, roc4)
# This is exactly equivalent:
power.roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(1, 0.9))
}
# With Obuchowski variance
power.roc.test(roc3, roc4, method="obuchowski")
## Significance level
# With bootstrap variance (default)
\dontrun{
power.roc.test(roc3, roc4, power=0.9, sig.level=NULL)
}
# With Obuchowski variance
power.roc.test(roc3, roc4, power=0.9, sig.level=NULL, method="obuchowski")
## With only binormal parameters given
# From example 2 of Obuchowski and McClish, 1997.
ob.params <- list(A1=2.6, B1=1, A2=1.9, B2=1, rn=0.6, ra=0.6, FPR11=0,
FPR12=0.2, FPR21=0, FPR22=0.2, delta=0.037)
power.roc.test(ob.params, power=0.8, sig.level=0.05)
power.roc.test(ob.params, power=0.8, sig.level=NULL, ncases=107)
power.roc.test(ob.params, power=NULL, sig.level=0.05, ncases=107)
}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{roc}
pROC/man/roc.Rd 0000644 0001762 0000144 00000040502 15040455005 012701 0 ustar ligges users \encoding{UTF-8}
\name{roc}
\alias{roc}
\alias{roc_}
\alias{roc.formula}
\alias{roc.data.frame}
\alias{roc.default}
\title{
Build a ROC curve
}
\description{
This is the main function of the pROC package. It builds a ROC
curve and returns a \dQuote{roc} object, a list of class
\dQuote{roc}. This object can be \code{print}ed, \code{plot}ted, or
passed to the functions \code{\link{auc}}, \code{\link{ci}},
\code{\link{smooth.roc}} and \code{\link{coords}}. Additionally, two
\code{roc} objects can be compared with \code{\link{roc.test}}.
}
\usage{
roc(...)
\S3method{roc}{formula}(formula, data, ...)
\S3method{roc}{data.frame}(data, response, predictor,
ret = c("roc", "coords", "all_coords"), ...)
\S3method{roc}{default}(response, predictor, controls, cases,
density.controls, density.cases,
levels=base::levels(as.factor(response)), percent=FALSE, na.rm=TRUE,
direction=c("auto", "<", ">"), algorithm = 2, quiet = FALSE,
smooth=FALSE, auc=TRUE, ci=FALSE, plot=FALSE, smooth.method="binormal",
smooth.n=512, ci.method=NULL, density=NULL, ...)
roc_(data, response, predictor, ret = c("roc", "coords", "all_coords"), ...)
}
\arguments{
\item{response}{a factor, numeric or character vector of
responses (true class), typically encoded with 0 (controls) and 1 (cases).
Only two classes can be used in a ROC curve. If the vector
contains more than two unique values, or if their order could be
ambiguous, use \code{levels} to specify which values must be used as
control and case value.
If the first argument was a \code{\link{data.frame}}, \code{response}
should be the name of the column in \code{data} containing the
response, quoted for \code{roc_}, and optionally quoted for
\code{roc.data.frame} (non-standard evaluation or NSE).
}
\item{predictor}{a \code{\link{numeric}} or \code{\link{ordered}} vector
of the same length than \code{response}, containing the predicted
value of each observation.
If the first argument was a \code{\link{data.frame}}, \code{predictor}
should be the name of the column in \code{data} containing the
predictor, quoted for \code{roc_}, and optionally quoted for
\code{roc.data.frame} (non-standard evaluation or NSE).
}
\item{controls, cases}{instead of \code{response}, \code{predictor},
the data can be supplied as two \code{\link{numeric}} or
\code{\link{ordered}} vectors containing the predictor
values for control and case observations.
}
\item{density.controls, density.cases}{a smoothed ROC curve can be
built directly from two densities on identical \code{x} points, as in
\code{\link[=smooth.roc]{smooth}}.
}
\item{formula, data}{a formula of the type \code{response~predictor}. If mulitple predictors
are passed, a named list of \code{roc} objects will be returned. Additional arguments
\code{data} and \code{subset}, but not \code{na.action} are supported, see
\code{\link{model.frame}} for more details.
}
\item{levels}{the value of the response for controls and cases
respectively. By default, the first two values of
\code{levels(as.factor(response))} are taken, and the remaining levels are ignored.
It usually captures two-class factor data correctly, but will
frequently fail for other data types (response factor with more than 2 levels,
or for example if your response is coded \dQuote{controls} and \dQuote{cases},
the levels will be inverted) and must then be specified here.
If your data is coded as \code{0} and \code{1} with \code{0}
being the controls, you can safely omit this argument.
}
\item{percent}{if the sensitivities, specificities and AUC must be
given in percent (\code{TRUE}) or in fraction (\code{FALSE}, default).
}
\item{na.rm}{if \code{TRUE}, the \code{NA} values will be removed
(ignored by \code{roc.formula}).
}
\item{direction}{how are positive observations defined?
\dQuote{<}: observations are positive when they
are greater than or equal (\code{>=}) to the threshold.
\dQuote{>}: observations are positive when they are smaller than or equal
(\code{<=}) to the threshold.
\dQuote{auto} (default): automatically detect in which group the
median is higher and take the direction accordingly. See details.
You should set this explicity to \dQuote{>} or \dQuote{<} whenever
you are resampling or randomizing the data, otherwise the
curves will be biased towards higher AUC values.
}
\item{algorithm}{DEPRECATED. A value other than \code{2} will produce a
warning. The argument will be removed in a future version.
}
\item{ret}{for \code{roc.data.frame} only, whether to return the
threshold sensitivity and specificity at all thresholds (\dQuote{coords}),
all the coordinates at all thresholds (\dQuote{all_coords}) or the
\code{roc} object (\dQuote{roc}).}
\item{quiet}{set to \code{TRUE} to turn off \code{\link{message}}s
when \code{direction} and \code{levels} are auto-detected.
}
\item{smooth}{if TRUE, the ROC curve is passed to \code{\link{smooth}}
to be smoothed.
}
\item{auc}{compute the area under the curve (AUC)? If \code{TRUE}
(default), additional arguments can be passed to \code{\link{auc}}.
}
\item{ci}{compute the confidence interval (CI)? If set to \code{TRUE},
additional arguments can be passed to \code{\link{ci}}.
}
\item{plot}{plot the ROC curve? If \code{TRUE}, additional
arguments can be passed to \code{\link{plot.roc}}.
}
\item{smooth.method, smooth.n, ci.method}{in \code{roc.formula} and
\code{roc.default}, the \code{method} and \code{n} arguments to
\code{\link[=smooth.roc]{smooth}} (if \code{smooth=TRUE}) and
\code{of="auc"}) must be passed as
\code{smooth.method}, \code{smooth.n} and \code{ci.method} to avoid confusions.
}
\item{density}{\code{density} argument passed to \code{\link[=smooth.roc]{smooth}}.}
\item{\dots}{further arguments passed to or from other methods, and
especially:
\itemize{
\item \code{\link{auc}}: \code{partial.auc}, \code{partial.auc.focus}, \code{partial.auc.correct}.
\item \code{\link{ci}}: \code{of}, \code{conf.level}, \code{boot.n}, \code{boot.stratified}, \code{progress}
\item \code{\link{ci.auc}}:, \code{reuse.auc}, \code{method}
\item \code{\link{ci.thresholds}}: \code{thresholds}
\item \code{\link{ci.se}}: \code{sensitivities}
\item \code{\link{ci.sp}}: \code{specificities}
\item \code{\link{plot.roc}}: \code{add}, \code{col} and most
other arguments to the \code{\link{plot.roc}} function. See
\code{\link{plot.roc}} directly for more details.
\item \code{\link{smooth}}: \code{method}, \code{n}, and all other
arguments. See \code{\link{smooth}} for more details.
}
}
}
\details{
This function's main job is to build a ROC object. See the
\dQuote{Value} section to this page for more details. Before
returning, it will call (in this order) the \code{\link[=smooth.roc]{smooth}},
\code{\link{auc}}, \code{\link{ci}} and \code{\link{plot.roc}}
functions if \code{smooth} \code{auc}, \code{ci} and \code{plot.roc}
(respectively) arguments are set to TRUE. By default, only \code{auc}
is called.
Data can be provided as \code{response, predictor}, where the
predictor is the numeric (or ordered) level of the evaluated signal, and
the response encodes the observation class (control or case). The
\code{level} argument specifies which response level must be taken as
controls (first value of \code{level}) or cases (second). It can
safely be ignored when the response is encoded as \code{0} and
\code{1}, but it will frequently fail otherwise. By default, the first
two values of \code{levels(as.factor(response))} are taken, and the
remaining levels are ignored. This means that if your response is
coded \dQuote{control} and \dQuote{case}, the levels will be
inverted.
In some cases, it is more convenient to pass the data as
\code{controls, cases}, but both arguments are ignored if
\code{response, predictor} was specified to non-\code{NULL} values.
It is also possible to pass density data with \code{density.controls,
density.cases}, which will result in a smoothed ROC curve even if
\code{smooth=FALSE}, but are ignored if \code{response, predictor} or
\code{controls, cases} are provided.
Thresholds are selected as the means between any two consecutive values
observed in the data. This choice is aimed to facilitate their interpretation,
as any data point will be unambiguously positive or negative
regardless of whether the comparison operator includes equality
or not. As a corollary, thresholds do not correspond to actual values
in the data.
Specifications for \code{\link{auc}}, \code{\link{ci}} and
\code{\link{plot.roc}} are not kept if \code{auc}, \code{ci} or \code{plot} are set to
\code{FALSE}. Especially, in the following case:
\preformatted{
myRoc <- roc(..., auc.polygon=TRUE, grid=TRUE, plot=FALSE)
plot(myRoc)
}
the plot will not have the AUC polygon nor the grid. Similarly, when
comparing \dQuote{roc} objects, the following is not possible:
\preformatted{
roc1 <- roc(..., partial.auc=c(1, 0.8), auc=FALSE)
roc2 <- roc(..., partial.auc=c(1, 0.8), auc=FALSE)
roc.test(roc1, roc2)
}
This will produce a test on the full AUC, not the partial AUC. To make
a comparison on the partial AUC, you must repeat the specifications
when calling \code{\link{roc.test}}:
\preformatted{
roc.test(roc1, roc2, partial.auc=c(1, 0.8))
}
Note that if \code{roc} was called with \code{auc=TRUE}, the latter syntax will not
allow redefining the AUC specifications. You must use \code{reuse.auc=FALSE} for that.
}
\value{
If the data contained any \code{NA} value and \code{na.rm=FALSE}, \code{NA} is
returned. Otherwise, if \code{smooth=FALSE}, a list of class
\dQuote{roc} with the following fields:
\item{auc}{if called with \code{auc=TRUE}, a numeric of class \dQuote{auc} as
defined in \code{\link{auc}}.
}
\item{ci}{if called with \code{ci=TRUE}, a numeric of class \dQuote{ci} as
defined in \code{\link{ci}}.
}
\item{response}{the response vector. Patients whose response is not
\code{\link{\%in\%}} \code{levels} are discarded. If \code{NA} values
were removed, a \code{na.action} attribute similar
to \code{\link{na.omit}} stores the row numbers.
}
\item{predictor}{the predictor vector converted to numeric as used to build the ROC
curve. Patients whose response is not \code{\link{\%in\%}} \code{levels} are discarded. If
\code{NA} values were removed, a \code{na.action} attribute similar
to \code{\link{na.omit}} stores the row numbers.
}
\item{original.predictor, original.response}{the response and predictor vectors as passed in argument.}
\item{levels}{the levels of the response as defined in argument.}
\item{controls}{the predictor values for the control observations.}
\item{cases}{the predictor values for the cases.}
\item{percent}{if the sensitivities, specificities and AUC are
reported in percent, as defined in argument.
}
\item{direction}{the direction of the comparison, as defined in argument.}
\item{sensitivities}{the sensitivities defining the ROC curve.}
\item{specificities}{the specificities defining the ROC curve.}
\item{thresholds}{the thresholds at which the sensitivities and
specificities were computed. See below for details.
}
\item{call}{how the function was called. See \code{\link{match.call}} for
more details.
}
\item{fun.sesp}{DEPRECATED. The value will be removed in a future version.}
If \code{smooth=TRUE} a list of class \dQuote{smooth.roc} as returned
by \code{\link{smooth}}, with or without additional elements
\code{auc} and \code{ci} (according to the call).
}
\section{Pipelines}{
The \code{roc} function can be used in pipelines, for instance with \pkg{dplyr} or \pkg{magrittr}.
The \code{roc.data.frame} method supports both standard and non-standard evaluation (NSE):
\preformatted{
library(dplyr)
# Standard evaluation:
aSAH \%>\%
filter(gender == "Female") \%>\%
roc("outcome", "s100b")
# Non-Standard Evaluation:
aSAH \%>\%
filter(gender == "Female") \%>\%
roc(outcome, s100b)
}
For tasks involving programming and variable column names, the \code{roc_} function provides
standard evaluation:
\preformatted{
# Standard evaluation:
aSAH \%>\%
filter(gender == "Female") \%>\%
roc_("outcome", "s100b")
}
By default it returns the \code{\link{roc}} object, which can then be piped to
the \code{\link{coords}} function to extract coordinates that can be used
in further pipelines.
\preformatted{
# Returns thresholds, sensitivities and specificities:
aSAH \%>\%
roc(outcome, s100b) \%>\%
coords(transpose = FALSE) \%>\%
filter(sensitivity > 0.6,
specificity > 0.6)
# Returns all existing coordinates, then select precision and recall:
aSAH \%>\%
roc(outcome, s100b) \%>\%
coords(ret = "all", transpose = FALSE) \%>\%
select(precision, recall)
}
}
\section{Errors}{
If no control or case observation exist for the given levels of
response, no ROC curve can be built and an error is triggered with
message \dQuote{No control observation} or \dQuote{No case
observation}.
If the predictor is not a numeric or ordered, as defined by
\code{\link{as.numeric}} or \code{\link{as.ordered}}, the message
\dQuote{Predictor must be numeric or ordered} is returned.
The message \dQuote{No valid data provided} is issued when the data
wasn't properly passed. Remember you need both \code{response} and
\code{predictor} of the same (not null) length, or both \code{controls}
and \code{cases}. Combinations such as \code{predictor} and
\code{cases} are not valid and will trigger this error.
Infinite values of the predictor cannot always be thresholded by
infinity and can cause ROC curves to not reach 0 or 100\%
specificity or sensitivity. Since version 1.13.0, pROC returns \code{NaN}
with a warning message \dQuote{Infinite value(s) in predictor} if
\code{predictor} contains any \link[=is.infinite]{infinite} values.
}
\references{
Tom Fawcett (2006) ``An introduction to ROC analysis''. \emph{Pattern
Recognition Letters} \bold{27}, 861--874. DOI:
\doi{10.1016/j.patrec.2005.10.010}.
Xavier Robin, Natacha Turck, Alexandre Hainard, \emph{et al.}
(2011) ``pROC: an open-source package for R and S+ to analyze and
compare ROC curves''. \emph{BMC Bioinformatics}, \bold{7}, 77.
DOI: \doi{10.1186/1471-2105-12-77}.
}
\seealso{
\code{\link{auc}}, \code{\link{ci}}, \code{\link{plot.roc}}, \code{\link{print.roc}}, \code{\link{roc.test}}
}
\examples{
data(aSAH)
# Basic example
roc(aSAH$outcome, aSAH$s100b,
levels=c("Good", "Poor"))
# As levels aSAH$outcome == c("Good", "Poor"),
# this is equivalent to:
roc(aSAH$outcome, aSAH$s100b)
# In some cases, ignoring levels could lead to unexpected results
# Equivalent syntaxes:
roc(outcome ~ s100b, aSAH)
roc(aSAH$outcome ~ aSAH$s100b)
with(aSAH, roc(outcome, s100b))
with(aSAH, roc(outcome ~ s100b))
# With a formula:
roc(outcome ~ s100b, data=aSAH)
\dontrun{
library(dplyr)
aSAH \%>\%
filter(gender == "Female") \%>\%
roc(outcome, s100b)
}
# Using subset (only with formula)
roc(outcome ~ s100b, data=aSAH, subset=(gender == "Male"))
roc(outcome ~ s100b, data=aSAH, subset=(gender == "Female"))
# With numeric controls/cases
roc(controls=aSAH$s100b[aSAH$outcome=="Good"], cases=aSAH$s100b[aSAH$outcome=="Poor"])
# With ordered controls/cases
roc(controls=aSAH$wfns[aSAH$outcome=="Good"], cases=aSAH$wfns[aSAH$outcome=="Poor"])
# Inverted the levels: "Poor" are now controls and "Good" cases:
roc(aSAH$outcome, aSAH$s100b,
levels=c("Poor", "Good"))
# The result was exactly the same because of direction="auto".
# The following will give an AUC < 0.5:
roc(aSAH$outcome, aSAH$s100b,
levels=c("Poor", "Good"), direction="<")
# If we are sure about levels and direction auto-detection,
# we can turn off the messages:
roc(aSAH$outcome, aSAH$s100b, quiet = TRUE)
# If we prefer counting in percent:
roc(aSAH$outcome, aSAH$s100b, percent=TRUE)
# Plot and CI (see plot.roc and ci for more options):
roc(aSAH$outcome, aSAH$s100b,
percent=TRUE, plot=TRUE, ci=TRUE)
# Smoothed ROC curve
roc(aSAH$outcome, aSAH$s100b, smooth=TRUE)
# this is not identical to
smooth(roc(aSAH$outcome, aSAH$s100b))
# because in the latter case, the returned object contains no AUC
}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{roc}
pROC/man/coords_transpose.Rd 0000644 0001762 0000144 00000006211 15040443562 015511 0 ustar ligges users \encoding{UTF-8}
\name{coords_transpose}
\alias{coords_transpose}
\title{Transposing the output of \code{coords}}
\description{This help page desribes recent and upcoming changes in the
return values of the \code{\link{coords}} function.}
\section{Background information}{
Until the release of pROC 1.16, the \code{coords} function was returning
a matrix with thresholds in columns, and the coordinate variables in rows.
\preformatted{
data(aSAH)
rocobj <- roc(aSAH$outcome, aSAH$s100b)
coords(rocobj, c(0.05, 0.2, 0.5))
# 0.05 0.2 0.5
# threshold 0.05000000 0.2000000 0.5000000
# specificity 0.06944444 0.8055556 0.9722222
# sensitivity 0.97560976 0.6341463 0.2926829
}
This format didn't conform to the grammar of the
\href{https://www.tidyverse.org/}{tidyverse} which has become prevalent in modern
R language.
In addition, the dropping of dimensions by default makes it difficult to guess
what type of data \code{coords} is going to return.
\preformatted{
coords(rocobj, "best")
# threshold specificity sensitivity
# 0.2050000 0.8055556 0.6341463
# A numeric vector
}
Although it is possible to pass \code{drop = FALSE}, the fact that it is not the
default makes the behaviour unintuitive.
In pROC version 1.16, this was changed and \code{coords} now returns
a \code{\link{data.frame}} with the thresholds in rows and measurement in colums by default.
\preformatted{
coords(rocobj, c(0.05, 0.2, 0.5), transpose = FALSE)
# threshold specificity sensitivity
# 0.05 0.05 0.06944444 0.9756098
# 0.2 0.20 0.80555556 0.6341463
# 0.5 0.50 0.97222222 0.2926829
}
}
\section{Changes in 1.15}{
\enumerate{
\item{Addition of the \code{transpose} argument.}
\item{Display a warning if \code{transpose} is missing. Pass \code{transpose} explicitly to silence the warning.}
\item{Deprecation of \code{as.list}.}
}
}
\section{Changes in 1.16}{
\itemize{
\item{Change of the default \code{transpose} to \code{TRUE}.}
}
THIS CHANGE IS BACKWARDS INCOMPATIBLE AND IS EXPECTED TO BREAK LEGACY CODE.
}
\section{Changes in 1.17}{
\itemize{
\item{Dropped the warning if \code{transpose} is missing.}
}
}
\section{Changes in 1.19}{
\enumerate{
\item{Setting \code{transpose=TRUE} is deprecated and triggers a warning.}
\item{The \code{as.list}, \code{as.matrix} and \code{drop} are deprecated.
Setting them to any value triggers a waring.}
\item{\code{transpose=FALSE} continues to work normally.}
}
}
\section{Changes in future versions}{
\enumerate{
\item{Setting \code{transpose} to \code{TRUE} will stop working and result
in an error.}
\item{The \code{drop}, \code{as.list} and \code{as.matrix} arguments will
be removed.}
\item{\code{transpose=FALSE} will keep working indefinitely.}
}
}
\section{Related changes in ci.coords}{
In version 1.16, the format of the \code{\link{ci.coords}} return value was changed from a matrix-like object with mixed \code{x} and \code{ret} in rows and 3 columns, into a list-like object which should be easier to use programatically.
}
\section{See also}{
\href{https://github.com/xrobin/pROC/issues/54}{The GitHub issue tracking the changes described in this manual page}.
}
pROC/man/var.Rd 0000644 0001762 0000144 00000023774 15040443562 012727 0 ustar ligges users \encoding{UTF-8}
\name{var.roc}
\alias{var}
\alias{var.roc}
\alias{var.auc}
\alias{var.smooth.roc}
\alias{var.default}
\title{
Variance of a ROC curve
}
\description{
These functions compute the variance of the AUC of a ROC curve.
}
\usage{
var(...)
\S3method{var}{default}(...)
\S3method{var}{auc}(auc, ...)
\S3method{var}{roc}(roc, method=c("delong", "bootstrap", "obuchowski"),
boot.n = 2000, boot.stratified = TRUE, reuse.auc=TRUE,
progress = NULL, parallel=FALSE, ...)
\S3method{var}{smooth.roc}(smooth.roc, ...)
}
\arguments{
\item{roc, smooth.roc, auc}{a \dQuote{roc} object from the
\code{\link{roc}} function, a \dQuote{smooth.roc} object from the
\code{\link[=smooth.roc]{smooth}} function or an \dQuote{auc} object from
the \code{\link{auc}} function.
}
\item{method}{the method to use, either \dQuote{delong} or
\dQuote{bootstrap}. The first letter is
sufficient. If omitted, the appropriate method is selected as
explained in details.
}
\item{reuse.auc}{if \code{TRUE} (default) and the \dQuote{roc} objects
contain an \dQuote{auc} field, re-use these specifications for the
test. See details.
}
\item{boot.n}{for \code{method="bootstrap"} only: the number of
bootstrap replicates or permutations. Default: \var{2000}.
}
\item{boot.stratified}{for \code{method="bootstrap"} only:
should the bootstrap be stratified (same number
of cases/controls in each replicate than in the original sample) or
not. Default: \var{TRUE}.
}
\item{progress}{DEPRECATED. A value other than \code{NULL} will produce
a warning. The argument will be removed in a future version.}
\item{parallel}{DEPRECATED. A value other than \code{FALSE} will produce a warning.
The argument will be removed in a future version.}
\item{\dots}{further arguments passed to or from other methods,
especially arguments for \code{var.roc} when calling \code{var},
\code{var.auc} and \code{var.smooth.roc}. Arguments for
\code{\link{auc}} (if \code{reuse.auc=FALSE}) and
\code{\link{txtProgressBar}} (only \code{char} and \code{style}) if
applicable.
}
}
\details{
The \code{var} function computes the variance of the AUC of a ROC
curve. It is typically called with the \link{roc} object of
interest. Two methods are available: \dQuote{delong} and
\dQuote{bootstrap} (see \dQuote{Computational
details} section below).
The default is to use \dQuote{delong} method except for with
partial AUC and smoothed curves where \dQuote{bootstrap} is employed.
Using \dQuote{delong} for partial AUC and smoothed ROCs is not
supported.
For \link[=smooth.roc]{smoothed ROC curves}, smoothing is performed again at each
bootstrap replicate with the parameters originally provided.
If a density smoothing was performed with user-provided
\code{density.cases} or \code{density.controls} the bootstrap cannot
be performed and an error is issued.
\code{var.default} forces the usage of the
\code{\link[stats:cor]{var}} function in the \pkg{stats} package, so
that other code relying on \code{var} should continue to function
normally.
}
\section{AUC specification}{
\code{var} needs a specification of the AUC to compute
the variance of the AUC of the ROC curve.
The specification is defined by:
\enumerate{
\item the \dQuote{auc} field in the \dQuote{\link{roc}} objects if
\code{reuse.auc} is set to \code{TRUE} (default)
\item passing the specification to \code{\link{auc}} with \dots
(arguments \code{partial.auc}, \code{partial.auc.correct} and
\code{partial.auc.focus}). In this case, you must ensure either that
the \code{\link{roc}} object do not contain an \code{auc} field (if
you called \code{\link{roc}} with \code{auc=FALSE}), or set
\code{reuse.auc=FALSE}.
}
If \code{reuse.auc=FALSE} the \code{\link{auc}} function will always
be called with \code{\dots} to determine the specification, even if
the \dQuote{\link{roc}} objects do contain an \code{auc} field.
As well if the \dQuote{\link{roc}} objects do not contain an \code{auc}
field, the \code{\link{auc}} function will always be called with
\code{\dots} to determine the specification.
Warning: if the roc object passed to roc.test contains an \code{auc}
field and \code{reuse.auc=TRUE}, \link{auc} is not called and
arguments such as \code{partial.auc} are silently ignored.
}
\section{Computation details}{
With \code{method="bootstrap"}, the processing is done as follow:
\enumerate{
\item \code{boot.n} bootstrap replicates are drawn from the
data. If \code{boot.stratified} is \var{TRUE}, each replicate contains
exactly the same number of controls and cases than the original
sample, otherwise if \var{FALSE} the numbers can vary.
\item for each bootstrap replicate, the AUC of the ROC curve
is computed and stored.
\item the variance of the resampled AUCs are computed and returned.
}
With \code{method="delong"}, the processing is done as described in
Hanley and Hajian-Tilaki (1997) using the algorithm by Sun and Xu (2014).
With \code{method="obuchowski"}, the processing is done as described
in Obuchowski and McClish (1997), Table 1 and Equation 4, p. 1530--1531. The
computation of \eqn{g} for partial area under the ROC curve is
modified as:
\deqn{expr1 * (2 * pi * expr2) ^ {(-1)} * (-expr4) - A * B * expr1 * (2 * pi * expr2^3) ^ {(-1/2)} * expr3}.
}
\section{Binormality assumption}{
The \dQuote{obuchowski} method makes the assumption that the data is binormal.
If the data shows a deviation from this assumption, it might help to
normalize the data first (that is, before calling \code{\link{roc}}),
for example with quantile normalization:
\preformatted{
norm.x <- qnorm(rank(x)/(length(x)+1))
var(roc(response, norm.x, ...), ...)
}
\dQuote{delong} and \dQuote{bootstrap} methods make no such assumption.
}
\value{
The numeric value of the variance.
}
\section{Warnings}{
If \code{method="delong"} and the AUC specification specifies a
partial AUC, the warning \dQuote{Using DeLong for partial AUC is
not supported. Using bootstrap test instead.} is issued. The
\code{method} argument is ignored and \dQuote{bootstrap} is used instead.
If \code{method="delong"} and the ROC
curve is smoothed, the warning \dQuote{Using DeLong for
smoothed ROCs is not supported. Using bootstrap test instead.} is
issued. The \code{method} argument is ignored and \dQuote{bootstrap}
is used instead.
If \code{boot.stratified=FALSE} and the sample has a large imbalance between
cases and controls, it could happen that one or more of the replicates
contains no case or control observation, or that there are not enough
points for smoothing, producing a \code{NA} area.
The warning \dQuote{NA value(s) produced during bootstrap were ignored.}
will be issued and the observation will be ignored. If you have a large
imbalance in your sample, it could be safer to keep
\code{boot.stratified=TRUE}.
When the ROC curve has an \code{\link{auc}} of 1 (or 100\%), the variance will always be null.
This is true for both \dQuote{delong} and \dQuote{bootstrap} methods that can
not properly assess the variance in this case. This result is misleading, as the variance is of course not null.
A \code{\link{warning}} will be displayed to inform of this condition, and of the misleading output.
}
\section{Errors}{
If \code{density.cases} and \code{density.controls} were provided
for smoothing, the error \dQuote{Cannot compute the covariance on ROC
curves smoothed with density.controls and density.cases.} is
issued.
}
\references{
Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson
(1988) ``Comparing the areas under two or more correlated receiver
operating characteristic curves: a nonparametric
approach''. \emph{Biometrics} \bold{44}, 837--845.
James A. Hanley and Karim O. Hajian-Tilaki (1997) ``Sampling
variability of nonparametric estimates of the areas under receiver
operating characteristic curves: An update''. \emph{Academic
Radiology} \bold{4}, 49--58. DOI:
\doi{10.1016/S1076-6332(97)80161-4}.
Nancy A. Obuchowski, Donna K. McClish (1997). ``Sample size
determination for diagnostic accurary studies involving binormal ROC
curve indices''. \emph{Statistics in Medicine}, \bold{16}(13),
1529--1542. DOI: \doi{10.1002/(SICI)1097-0258(19970715)16:13<1529::AID-SIM565>3.0.CO;2-H}.
Xu Sun and Weichao Xu (2014) ``Fast Implementation of DeLongs Algorithm for Comparing
the Areas Under Correlated Receiver Operating Characteristic Curves''. \emph{IEEE Signal
Processing Letters}, \bold{21}, 1389--1393.
DOI: \doi{10.1109/LSP.2014.2337313}.
Hadley Wickham (2011) ``The Split-Apply-Combine Strategy for Data Analysis''. \emph{Journal of Statistical Software}, \bold{40}, 1--29.
URL: \doi{10.18637/jss.v040.i01}.
}
\seealso{
\code{\link{roc}}, \code{\link{cov.roc}}
}
\examples{
data(aSAH)
## Basic example
roc1 <- roc(aSAH$outcome, aSAH$s100b)
roc2 <- roc(aSAH$outcome, aSAH$wfns)
var(roc1)
var(roc2)
# We could also write it in one line:
var(roc(aSAH$outcome, aSAH$s100b))
\dontrun{
# The latter used Delong. To use bootstrap:
var(roc1, method="bootstrap")
# Decrease boot.n for a faster execution
var(roc1,method="bootstrap", boot.n=1000)
}
# To use obuchowski:
var(roc1, method="obuchowski")
\dontrun{
# Variance of smoothed ROCs:
# Smoothing is re-done at each iteration, and execution is slow
var(smooth(roc1))
}
# or from an AUC (no smoothing)
var(auc(roc1))
## Test data from Hanley and Hajian-Tilaki, 1997
disease.present <- c("Yes", "No", "Yes", "No", "No", "Yes", "Yes", "No",
"No", "Yes", "No", "No", "Yes", "No", "No")
field.strength.1 <- c(1, 2, 5, 1, 1, 1, 2, 1, 2, 2, 1, 1, 5, 1, 1)
field.strength.2 <- c(1, 1, 5, 1, 1, 1, 4, 1, 2, 2, 1, 1, 5, 1, 1)
roc3 <- roc(disease.present, field.strength.1)
roc4 <- roc(disease.present, field.strength.2)
# Assess the variance:
var(roc3)
var(roc4)
\dontrun{
# With bootstrap:
var(roc3, method="bootstrap")
var(roc4, method="bootstrap")
}
}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{roc}
pROC/man/print.Rd 0000644 0001762 0000144 00000004633 13607143106 013262 0 ustar ligges users \encoding{UTF-8}
\name{print}
\alias{print.roc}
\alias{print.smooth.roc}
\alias{print.multiclass.roc}
\alias{print.mv.multiclass.roc}
\alias{print.ci.auc}
\alias{print.ci.thresholds}
\alias{print.ci.coords}
\alias{print.ci.se}
\alias{print.ci.sp}
\alias{print.auc}
\alias{print.multiclass.auc}
\title{
Print a ROC curve object
}
\description{
This function prints a ROC curve, AUC or CI object and return it invisibly.
}
\usage{
\S3method{print}{roc}(x, digits=max(3, getOption("digits") - 3), call=TRUE, ...)
\S3method{print}{multiclass.roc}(x, digits=max(3, getOption("digits") -
3), call=TRUE, ...)
\S3method{print}{mv.multiclass.roc}(x, digits=max(3, getOption("digits") -
3), call=TRUE, ...)
\S3method{print}{smooth.roc}(x, digits=max(3, getOption("digits") - 3),
call=TRUE, ...)
\S3method{print}{auc}(x, digits=max(3, getOption("digits") - 3), ...)
\S3method{print}{multiclass.auc}(x, digits=max(3, getOption("digits") - 3), ...)
\S3method{print}{ci.auc}(x, digits=max(3, getOption("digits") - 3), ...)
\S3method{print}{ci.thresholds}(x, digits=max(3, getOption("digits") - 3), ...)
\S3method{print}{ci.se}(x, digits=max(3, getOption("digits") - 3), ...)
\S3method{print}{ci.sp}(x, digits=max(3, getOption("digits") - 3), ...)
\S3method{print}{ci.coords}(x, digits=max(3, getOption("digits") - 3), ...)
}
\arguments{
\item{x}{a roc, auc or ci object, from the \link{roc}, \link{auc} or
\link{ci} functions respectively.
}
\item{call}{if the call is printed.}
\item{digits}{the number of significant figures to print. See
\link{signif} for more details.
}
\item{\dots}{further arguments passed to or from other methods. In
particular, \code{print.roc} calls \code{print.auc} and the
\code{print.ci} variants internally, and a \code{digits} argument is
propagated. Not used in print.auc and print.ci variants.
}
}
\value{
These functions return the object they were passed invisibly.
}
\seealso{
\code{\link{roc}}, \code{\link{auc}}, \code{\link{ci}}, \code{\link{coords}}
}
\examples{
data(aSAH)
# Print a roc object:
rocobj <- roc(aSAH$outcome, aSAH$s100b)
print(rocobj)
# Print a smoothed roc object
print(smooth(rocobj))
# implicit printing
roc(aSAH$outcome, aSAH$s100b)
# Print an auc and a ci object, from the ROC object or calling
# the dedicated function:
print(rocobj$auc)
print(ci(rocobj))
}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{print}
\keyword{roc}
pROC/man/groupGeneric.Rd 0000644 0001762 0000144 00000002736 13607143106 014561 0 ustar ligges users \encoding{UTF-8}
\name{groupGeneric}
\alias{groupGeneric}
\alias{groupGeneric.pROC}
\alias{groupGeneric.auc}
\alias{groupGeneric.ci.coords}
\alias{groupGeneric.ci.se}
\alias{groupGeneric.ci.sp}
\alias{Ops}
\alias{Math}
\alias{Ops.auc}
\alias{Math.auc}
\alias{Ops.ci.coords}
\alias{Math.ci.coords}
\alias{Ops.ci.se}
\alias{Math.ci.se}
\alias{Ops.ci.sp}
\alias{Math.ci.sp}
\alias{Ops.ci.auc}
\alias{Math.ci.auc}
\alias{Ops.ci}
\alias{Math.ci}
\title{
pROC Group Generic Functions
}
\description{
Redefine \pkg{base} groupGeneric functions to handle \code{\link{auc}} and \code{\link{ci}} objects properly on operations and mathematical operations.
Attributes are dropped so that the AUC/CI behaves as a numeric value/matrix, respectively.
In the case of AUC, all attributes are dropped, while in CI only the CI-specific attributes are, keeping those necessary for the matrices.
}
\usage{
\special{Math(x, \dots)}
\special{Ops(e1, e2)}
}
\arguments{
\item{x, e1, e2}{\code{\link{auc}} objects, or mixed numerics and \code{auc} objects.
}
\item{\dots}{further arguments passed to other Math methods.
}
}
\seealso{
\code{\link{groupGeneric}}, \code{\link{auc}}
}
\examples{
data(aSAH)
# Create a roc object:
aucobj1 <- auc(roc(aSAH$outcome, aSAH$s100b))
aucobj2 <- auc(roc(aSAH$outcome, aSAH$wfns))
# Math
sqrt(aucobj1)
round(aucobj2, digits=1)
# Ops
aucobj1 * 2
2 * aucobj2
aucobj1 + aucobj2
# With CI
ciaucobj <- ci(aucobj1)
ciaucobj * 2
sqrt(ciaucobj)
}
\keyword{methods}
pROC/man/ci.se.Rd 0000644 0001762 0000144 00000014421 15040443562 013125 0 ustar ligges users \encoding{UTF-8}
\name{ci.se}
\alias{ci.se}
\alias{ci.se.default}
\alias{ci.se.formula}
\alias{ci.se.roc}
\alias{ci.se.smooth.roc}
\title{
Compute the confidence interval of sensitivities at given specificities
}
\description{
This function computes the confidence interval (CI) of the sensitivity
at the given specificity points.
}
\usage{
# ci.se(...)
\S3method{ci.se}{roc}(roc, specificities = seq(0, 1, .1) * ifelse(roc$percent,
100, 1), conf.level=0.95, boot.n=2000, boot.stratified=TRUE,
progress = NULL, parallel=FALSE, ...)
\S3method{ci.se}{smooth.roc}(smooth.roc, specificities = seq(0, 1, .1) *
ifelse(smooth.roc$percent, 100, 1), conf.level=0.95, boot.n=2000,
boot.stratified=TRUE, progress = NULL,
parallel=FALSE, ...)
\S3method{ci.se}{formula}(formula, data, ...)
\S3method{ci.se}{default}(response, predictor, ...)
}
\arguments{
\item{roc, smooth.roc}{a \dQuote{roc} object from the
\code{\link{roc}} function, or a \dQuote{smooth.roc} object from the
\code{\link[=smooth.roc]{smooth}} function.
}
\item{response, predictor}{arguments for the \code{\link{roc}} function.}
\item{formula, data}{a formula (and possibly a data object) of type
response~predictor for the \code{\link{roc}} function.
}
\item{specificities}{on which specificities to evaluate the CI.
}
\item{conf.level}{the width of the confidence interval as [0,1], never
in percent. Default: 0.95, resulting in a 95\% CI.
}
\item{boot.n}{the number of bootstrap replicates. Default: 2000.}
\item{boot.stratified}{should the bootstrap be stratified (default, same number
of cases/controls in each replicate than in the original sample) or
not.
}
\item{progress}{DEPRECATED. A value other than \code{NULL} will produce
a warning. The argument will be removed in a future version.}
\item{parallel}{DEPRECATED. A value other than \code{FALSE} will produce a warning.
The argument will be removed in a future version.}
\item{\dots}{further arguments passed to or from other methods,
especially arguments for \code{\link{roc}} and \code{ci.se.roc}
when calling \code{ci.se.default} or \code{ci.se.formula}.
Arguments for \code{\link{txtProgressBar}} (only
\code{char} and \code{style}) if applicable.
}
}
\details{
\code{ci.se.formula} and \code{ci.se.default} are convenience methods
that build the ROC curve (with the \code{\link{roc}} function) before
calling \code{ci.se.roc}. You can pass them arguments for both
\code{\link{roc}} and \code{ci.se.roc}. Simply use \code{ci.se}
that will dispatch to the correct method.
The \code{ci.se.roc} function creates \code{boot.n} bootstrap replicate of the ROC
curve, and evaluates the sensitivity at specificities
given by the \code{specificities} argument. Then it computes the
confidence interval as the percentiles given by \code{conf.level}.
For more details about the bootstrap, see the Bootstrap section in
\link[=pROC-package]{this package's documentation}.
For \link[=smooth.roc]{smoothed ROC curves}, smoothing is performed again at each
bootstrap replicate with the parameters originally provided.
If a density smoothing was performed with user-provided
\code{density.cases} or \code{density.controls} the bootstrap cannot
be performed and an error is issued.
}
\section{Warnings}{
If \code{boot.stratified=FALSE} and the sample has a large imbalance between
cases and controls, it could happen that one or more of the replicates
contains no case or control observation, or that there are not enough
points for smoothing, producing a \code{NA} area.
The warning \dQuote{NA value(s) produced during bootstrap were ignored.}
will be issued and the observation will be ignored. If you have a large
imbalance in your sample, it could be safer to keep
\code{boot.stratified=TRUE}.
}
\section{Errors}{
If \code{density.cases} and \code{density.controls} were provided
for smoothing, the error \dQuote{Cannot compute the statistic on ROC
curves smoothed with density.controls and density.cases.} is issued.
}
\value{
A matrix of class \dQuote{ci.se}, \dQuote{ci} and \dQuote{matrix} (in this order)
containing the given sensitivities. Row (names) are the
specificities, the first column the lower bound, the 2nd column the
median and the 3rd column the upper bound.
Additionally, the list has the following attributes:
\item{conf.level}{the width of the CI, in fraction.}
\item{boot.n}{the number of bootstrap replicates.}
\item{boot.stratified}{whether or not the bootstrapping was stratified.}
\item{specificities}{the specificities as given in argument.}
\item{roc}{the object of class \dQuote{\link{roc}} that was used to
compute the CI.
}
}
\references{
James Carpenter and John Bithell (2000) ``Bootstrap condence intervals:
when, which, what? A practical guide for medical statisticians''.
\emph{Statistics in Medicine} \bold{19}, 1141--1164.
DOI: \doi{10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F}.
Tom Fawcett (2006) ``An introduction to ROC analysis''. \emph{Pattern
Recognition Letters} \bold{27}, 861--874. DOI:
\doi{10.1016/j.patrec.2005.10.010}.
Xavier Robin, Natacha Turck, Alexandre Hainard, \emph{et al.}
(2011) ``pROC: an open-source package for R and S+ to analyze and
compare ROC curves''. \emph{BMC Bioinformatics}, \bold{7}, 77.
DOI: \doi{10.1186/1471-2105-12-77}.
Hadley Wickham (2011) ``The Split-Apply-Combine Strategy for Data Analysis''. \emph{Journal of Statistical Software}, \bold{40}, 1--29.
URL: \doi{10.18637/jss.v040.i01}.
}
\seealso{
\code{\link{roc}},
\code{\link{ci}},
\code{\link{ci.sp}},
\code{\link{plot.ci}}
}
\examples{
# Create a ROC curve:
data(aSAH)
roc1 <- roc(aSAH$outcome, aSAH$s100b)
## Basic example ##
\dontrun{
ci.se(roc1)}\dontshow{ci.se(roc1, boot.n = 10)}
## More options ##
# Customized bootstrap and specificities:
\dontrun{
ci.se(roc1, c(.95, .9, .85), boot.n=10000, conf.level=0.9, stratified=FALSE)}\dontshow{
ci.se(roc1, c(.95, .9, .85), boot.n=10, conf.level=0.9, stratified=FALSE)}
## Plotting the CI ##
ci1 <- ci.se(roc1, boot.n = 10)
plot(roc1)
plot(ci1)
## On smoothed ROC curves with bootstrap ##
\dontrun{
ci.se(smooth(roc1, method="density"))}\dontshow{
ci.se(smooth(roc1, method="density"), boot.n = 10)}
}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{roc}
pROC/man/pROC-package.Rd 0000644 0001762 0000144 00000033225 15040443562 014323 0 ustar ligges users \encoding{UTF-8}
\name{pROC-package}
\alias{pROC-package}
\alias{pROC}
\docType{package}
\title{pROC}
\description{
Tools for visualizing, smoothing and comparing receiver operating
characteristic (ROC curves). (Partial) area under the curve (AUC) can
be compared with statistical tests based on U-statistics or
bootstrap. Confidence intervals can be computed for (p)AUC or ROC
curves. Sample size / power computation for one or two ROC curves are available.
}
\details{
The basic unit of the pROC package is the \code{\link{roc}} function. It
will build a ROC curve, smooth it if requested (if \code{smooth=TRUE}),
compute the AUC (if \code{auc=TRUE}), the confidence interval (CI) if
requested (if \code{ci=TRUE}) and plot the curve if requested (if
\code{plot=TRUE}).
The \code{\link{roc}} function will call \code{\link[=smooth.roc]{smooth}},
\code{\link{auc}},
\code{\link{ci}} and \code{\link{plot}} as necessary. See these
individual functions for the arguments that can be passed to them
through \code{\link{roc}}. These function can be called separately.
Two paired (that is \code{\link{roc}} objects with the same
\code{response}) or unpaired (with different \code{response}) ROC
curves can be compared with the \code{\link{roc.test}} function.
}
\section{Citation}{
If you use pROC in published research, please cite the following paper:
Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti,
Frédérique Lisacek, Jean-Charles Sanchez and Markus Müller (2011).
``pROC: an open-source package for R and S+ to analyze and compare ROC
curves''. \emph{BMC Bioinformatics}, \bold{12}, p. 77. DOI:
\doi{10.1186/1471-2105-12-77}
Type \code{citation("pROC")} for a BibTeX entry.
The authors would be glad to hear how pROC is employed. You are kindly
encouraged to notify Xavier Robin
about any work you publish.
}
\section{Abbreviations}{
The following abbreviations are employed extensively in this package:
\itemize{
\item ROC: receiver operating characteristic
\item AUC: area under the ROC curve
\item pAUC: partial area under the ROC curve
\item CI: confidence interval
\item SP: specificity
\item SE: sensitivity
}
}
\section{Functions}{
\tabular{ll}{
\code{\link{roc}} \tab Build a ROC curve\cr
\code{\link{are.paired}} \tab Dertermine if two ROC curves are paired \cr
\code{\link{auc}} \tab Compute the area under the ROC curve \cr
\code{\link{ci}} \tab Compute confidence intervals of a ROC curve \cr
\code{\link{ci.auc}} \tab Compute the CI of the AUC \cr
\code{\link{ci.coords}} \tab Compute the CI of arbitrary coordinates \cr
\code{\link{ci.se}} \tab Compute the CI of sensitivities at given specificities \cr
\code{\link{ci.sp}} \tab Compute the CI of specificities at given sensitivities \cr
\code{\link{ci.thresholds}} \tab Compute the CI of specificity and sensitivity of thresholds \cr
\code{\link{ci.coords}} \tab Compute the CI of arbitrary coordinates \cr
\code{\link{coords}} \tab Coordinates of the ROC curve \cr
\code{\link[=cov.roc]{cov}} \tab Covariance between two AUCs\cr
\code{\link{ggroc}} \tab Plot a ROC curve with \pkg{ggplot2}\cr
\code{\link{has.partial.auc}} \tab Determine if the ROC curve have a partial AUC\cr
\code{\link{lines.roc}} \tab Add a ROC line to a ROC plot \cr
\code{\link{plot.ci}} \tab Plot CIs \cr
\code{\link[=plot.roc]{plot}} \tab Plot a ROC curve \cr
\code{\link{power.roc.test}} \tab Sample size and power computation \cr
\code{\link[=print.roc]{print}} \tab Print a ROC curve object \cr
\code{\link{roc.test}} \tab Compare two ROC curves \cr
\code{\link[=smooth.roc]{smooth}} \tab Smooth a ROC curve \cr
\code{\link[=var.roc]{var}} \tab Variance of the AUC
}
}
\section{Dataset}{
This package comes with a dataset of 141 patients with aneurysmal
subarachnoid hemorrhage: \code{\link{aSAH}}.
}
\section{Installing and using}{
To install this package, make sure you are connected to the internet and issue the following command in the R prompt:
\preformatted{
install.packages("pROC")
}
To load the package in R:
\preformatted{
library(pROC)
}
}
\section{Experimental: pipelines}{
Since version 1.15.0, the \code{\link{roc}} function can be used in pipelines, for instance with \pkg{dplyr} or \pkg{magrittr}. This is still a highly experimental feature and will change significantly in future versions (see \href{https://github.com/xrobin/pROC/issues/54}{issue 54}).
The \code{\link{roc.data.frame}} method supports both standard and non-standard evaluation (NSE), and the \code{\link{roc_}}
function supports standard evaluation only.
\preformatted{
library(dplyr)
aSAH \%>\%
filter(gender == "Female") \%>\%
roc(outcome, s100b)
}
By default it returns the \code{\link{roc}} object, which can then be piped to
the \code{\link{coords}} function to extract coordinates that can be used
in further pipelines.
\preformatted{
aSAH \%>\%
filter(gender == "Female") \%>\%
roc(outcome, s100b) \%>\%
coords(transpose=FALSE) \%>\%
filter(sensitivity > 0.6,
specificity > 0.6)
}
More details and use cases are available in the \code{\link{roc}} help page.
}
\section{Bootstrap}{
All the bootstrap operations for \link[=roc.test]{significance testing}, \link[=ci]{confidence interval}, \link[=var]{variance} and \link[=cov]{covariance} computation are performed with non-parametric stratified or non-stratified resampling (according to the \code{stratified} argument) and with the percentile method, as described in Carpenter and Bithell (2000) sections 2.1 and 3.3.
Stratification of bootstrap can be controlled
with \code{boot.stratified}. In stratified bootstrap (the default), each replicate
contains the same number of cases and controls than the original
sample. Stratification is especially useful if one group has only
little observations, or if groups are not balanced.
The number of bootstrap replicates is controlled by \code{boot.n}. Higher numbers will give a more precise estimate of the significance tests and confidence intervals
but take more time to compute. 2000 is recommanded by Carpenter and Bithell (2000) for confidence intervals. In our experience this is sufficient for a good estimation of the
first significant digit only, so we recommend the use of 10000 bootstrap replicates to obtain a good estimate of the second significant digit whenever possible.
\subsection{Progress bars}{
Progress bars are no longer supported in pROC 1.19. The use of the
\code{progress} argument is deprecated, and any value other than \code{NULL}
will produce a warning. The argument will be removed in a future version.
}
}
\section{Handling large datasets}{
\subsection{Algorithms}{
Over the years, a significant amount of time has been invested in making pROC run faster and faster.
From the naive algorithm iterating over all thresholds implemented in the first version, we went to a
C++ implementation (with \pkg{Rcpp}), and a different algorithm using cummulative sum of responses sorted
by the predictor, which scales only with the number of data points, independently on the number of thresholds (\code{algorithm = 2}).
The curves themselves are identical, but computation time has been decreased massively.
Earlier versions of \pkg{pROC} allowed choosing the optimal algorithm. Since pROC 1.19, only a single
algorithm (previously \code{algorithm = 2}) is available. This allowed us to remove legacy code
and dependencies, and make \pkg{pROC} more maintainable and future proof.
}
\subsection{Boostrap}{
Bootstrap is typically slow because it involves repeatedly computing the ROC curve (or a part of it).
Some bootstrap functions are faster than others. Typically, \code{\link{ci.thresholds}} is the fastest, and \code{\link{ci.coords}} the slowest. Use \code{\link{ci.coords}} only if the CI you need cannot be computed by the specialized CI functions \code{\link{ci.thresholds}}, \code{\link{ci.se}} and \code{\link{ci.sp}}. Note that \code{\link{ci.auc}} cannot be replaced anyway.
A naive way to speed-up the boostrap is by removing the progress bar:
\preformatted{
rocobj <- roc(response, round(predictor))
system.time(ci(rocobj))
system.time(ci(rocobj, progress = "none"))
}
It is of course possible to reduce the number of boostrap iterations. See the \code{boot.n} argument to \code{\link{ci}}. This will reduce the precision of the bootstrap estimate.
\subsection{Parallel processing}{
Parallel processing is no longer supported in pROC 1.19. The use of the
\code{parallel} argument is deprecated, and any value other than
\code{FALSE} will produce a warning. The argument will be removed in a
future version.
}
}
\subsection{Using DeLong instead of boostrap}{
DeLong is an asymptotically exact method to evaluate the uncertainty of an AUC (DeLong \emph{et al.} (1988)). Since version 1.9, pROC uses the algorithm proposed by Sun and Xu (2014) which has an O(N log N) complexity and is always faster than bootstrapping. By default, pROC will choose the DeLong method whenever possible.
\preformatted{
rocobj <- roc(response, round(predictor), algorithm=3)
system.time(ci(rocobj, method="delong"))
}
}
}
\author{
Xavier Robin, Natacha Turck, Jean-Charles Sanchez and Markus Müller
Maintainer: Xavier Robin
}
\references{
James Carpenter and John Bithell (2000) ``Bootstrap condence intervals:
when, which, what? A practical guide for medical statisticians''.
\emph{Statistics in Medicine} \bold{19}, 1141--1164.
DOI: \doi{10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F}.
Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson
(1988) ``Comparing the areas under two or more correlated receiver
operating characteristic curves: a nonparametric
approach''. \emph{Biometrics} \bold{44}, 837--845.
Tom Fawcett (2006) ``An introduction to ROC analysis''. \emph{Pattern
Recognition Letters} \bold{27}, 861--874. DOI:
\doi{10.1016/j.patrec.2005.10.010}.
Xavier Robin, Natacha Turck, Alexandre Hainard, \emph{et al.}
(2011) ``pROC: an open-source package for R and S+ to analyze and
compare ROC curves''. \emph{BMC Bioinformatics}, \bold{7}, 77.
DOI: \doi{10.1186/1471-2105-12-77}.
Xu Sun and Weichao Xu (2014) ``Fast Implementation of DeLongs Algorithm for Comparing
the Areas Under Correlated Receiver Operating Characteristic Curves''. \emph{IEEE Signal
Processing Letters}, \bold{21}, 1389--1393.
DOI: \doi{10.1109/LSP.2014.2337313}.
Hadley Wickham (2011) ``The Split-Apply-Combine Strategy for Data Analysis''. \emph{Journal of Statistical Software}, \bold{40}, 1--29.
URL: \doi{10.18637/jss.v040.i01}.
}
\seealso{
CRAN packages \pkg{ROCR}, \pkg{verification} or Bioconductor's \pkg{roc}
for ROC curves.
CRAN packages \pkg{MASS} and \pkg{logcondens} employed in this package.
}
\examples{
data(aSAH)
## Build a ROC object and compute the AUC ##
roc1 <- roc(aSAH$outcome, aSAH$s100b)
print(roc1)
# With a formula
roc(outcome ~ s100b, aSAH)
# With pipes, dplyr-style:
\dontrun{
library(dplyr)
aSAH \%>\% roc(outcome, s100b)}
# Create a few more curves for the next examples
roc2 <- roc(aSAH$outcome, aSAH$wfns)
roc3 <- roc(aSAH$outcome, aSAH$ndka)
## AUC ##
auc(roc1, partial.auc = c(1, .9))
## Smooth ROC curve ##
smooth(roc1)
## Summary statistics
var(roc1)
cov(roc1, roc3)
## Plot the curve ##
plot(roc1)
# More plotting options, CI and plotting
# with all-in-one syntax:
roc4 <- roc(aSAH$outcome,
aSAH$s100b, percent=TRUE,
# arguments for auc
partial.auc=c(100, 90), partial.auc.correct=TRUE,
partial.auc.focus="sens",
# arguments for ci
ci=TRUE, boot.n=100, ci.alpha=0.9, stratified=FALSE,
# arguments for plot
plot=TRUE, auc.polygon=TRUE, max.auc.polygon=TRUE, grid=TRUE,
print.auc=TRUE, show.thres=TRUE)
# Add to an existing plot. Beware of 'percent' specification!
roc5 <- roc(aSAH$outcome, aSAH$wfns,
plot=TRUE, add=TRUE, percent=roc4$percent)
## With ggplot2 ##
if (require(ggplot2)) {
# Create multiple curves to plot
rocs <- roc(outcome ~ wfns + s100b + ndka, data = aSAH)
ggroc(rocs)
}
## Coordinates of the curve ##
coords(roc1, "best", ret=c("threshold", "specificity", "1-npv"))
coords(roc2, "local maximas", ret=c("threshold", "sens", "spec", "ppv", "npv"))
## Confidence intervals ##
# CI of the AUC
ci(roc2)
\dontrun{
# CI of the curve
sens.ci <- ci.se(roc1, specificities=seq(0, 100, 5))
plot(sens.ci, type="shape", col="lightblue")
plot(sens.ci, type="bars")}
# need to re-add roc2 over the shape
plot(roc2, add=TRUE)
\dontrun{
# CI of thresholds
plot(ci.thresholds(roc2))}
## Comparisons ##
# Test on the whole AUC
roc.test(roc1, roc2, reuse.auc=FALSE)
\dontrun{
# Test on a portion of the whole AUC
roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(100, 90),
partial.auc.focus="se", partial.auc.correct=TRUE)
# With modified bootstrap parameters
roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(100, 90),
partial.auc.correct=TRUE, boot.n=1000, boot.stratified=FALSE)}
## Power & sample size ##
# Power
# 1 curve
power.roc.test(roc1)
# 2 curves
power.roc.test(roc3, roc2)
# Sample size
# 1 curve
power.roc.test(roc3, power = 0.9)
# 2 curves
power.roc.test(roc1, roc2, power = 0.9)
# Also without ROC objects.
# For instance what AUC would be significantly different from 0.5?
power.roc.test(ncases=41, ncontrols=72, sig.level=0.05, power=0.95)
}
\keyword{package}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{print}
\keyword{htest}
\keyword{aplot}
\keyword{hplot}
\keyword{roc}
pROC/man/auc.Rd 0000644 0001762 0000144 00000017255 14114130125 012671 0 ustar ligges users \encoding{UTF-8}
\name{auc}
\alias{auc}
\alias{auc.default}
\alias{auc.formula}
\alias{auc.roc}
\alias{auc.smooth.roc}
\alias{auc.multiclass.roc}
\title{
Compute the area under the ROC curve
}
\description{
This function computes the numeric value of area under the ROC curve
(AUC) with the trapezoidal rule. Two syntaxes are possible: one object of class \dQuote{\link{roc}}, or either
two vectors (response, predictor) or a formula (response~predictor) as
in the \code{\link{roc}} function.
By default, the total AUC is computed, but a portion of the ROC curve
can be specified with \code{partial.auc}.
}
\usage{
auc(...)
\S3method{auc}{roc}(roc, partial.auc=FALSE, partial.auc.focus=c("specificity",
"sensitivity"), partial.auc.correct=FALSE,
allow.invalid.partial.auc.correct = FALSE, ...)
\S3method{auc}{smooth.roc}(smooth.roc, ...)
\S3method{auc}{multiclass.roc}(multiclass.roc, ...)
\S3method{auc}{formula}(formula, data, ...)
\S3method{auc}{default}(response, predictor, ...)
}
\arguments{
\item{roc, smooth.roc, multiclass.roc}{a \dQuote{roc} object from the
\code{\link{roc}} function, a \dQuote{smooth.roc} object from the
\code{\link[=smooth.roc]{smooth}} function, or a \dQuote{multiclass.roc}
or \dQuote{mv.multiclass.roc} from the \code{\link{multiclass.roc}} function.
}
\item{response, predictor}{arguments for the \code{\link{roc}} function.}
\item{formula, data}{a formula (and possibly a data object) of type response~predictor for the \code{\link{roc}} function.}
\item{partial.auc}{either \code{FALSE} (default: consider total area) or a
numeric vector of length 2: boundaries of the AUC to consider in
[0,1] (or [0,100] if percent is \code{TRUE}).
}
\item{partial.auc.focus}{if \code{partial.auc} is not \code{FALSE} and a partial
AUC is computed, specifies if \code{partial.auc} specifies the bounds in
terms of specificity (default) or sensitivity. Can be shortened to spec/sens
or even sp/se. Ignored if \code{partial.auc=FALSE}.
}
\item{partial.auc.correct}{logical indicating if the correction of
AUC must be applied in order to have a maximal AUC of 1.0 and a
non-discriminant AUC of 0.5 whatever the \code{partial.auc}
defined. Ignored if \code{partial.auc=FALSE}. Default: \code{FALSE}.
}
\item{allow.invalid.partial.auc.correct}{logical indicating if
the correction must return \code{NA} (with a \code{\link{warning}})
when attempting to correct a pAUC below the diagonal.
Set to \code{TRUE} to return a (probably invalid) corrected AUC.
This is useful especially to avoid introducing a bias against low
pAUCs in bootstrap operations.}
\item{\dots}{further arguments passed to or from other methods,
especially arguments for \code{\link{roc}} when calling
\code{auc.default}, \code{auc.formula}, \code{auc.smooth.roc}.
Note that the \code{auc}
argument of \code{\link{roc}} is not allowed. Unused in
\code{auc.roc}.
}
}
\details{
This function is typically called from \code{\link{roc}} when \code{auc=TRUE}
(default). It is also used by \code{\link{ci}}. When it is called with
two vectors (response, predictor) or a formula (response~predictor)
arguments, the \code{\link{roc}} function is called and only the AUC is
returned.
By default the total area under the curve is computed, but a partial AUC (pAUC)
can be
specified with the \code{partial.auc} argument. It specifies the bounds of
specificity or sensitivity (depending on \code{partial.auc.focus}) between
which the AUC will be computed. As it specifies specificities or
sensitivities, you must adapt it in relation to the 'percent'
specification (see details in \code{\link{roc}}).
\code{partial.auc.focus} is ignored if
\code{partial.auc=FALSE} (default). If a partial AUC is computed,
\code{partial.auc.focus} specifies if the bounds specified in
\code{partial.auc} must be interpreted as sensitivity or
specificity. Any other value will produce an error. It is recommended to
\code{\link[=plot.roc]{plot}} the ROC curve with \code{auc.polygon=TRUE} in order to
make sure the specification is correct.
If a pAUC is defined, it can be standardized (corrected). This correction is
controled by the \code{partial.auc.correct} argument. If \code{partial.auc.correct=TRUE},
the correction by McClish will be applied:
\deqn{\frac{1+\frac{auc-min}{max-min}}{2}}{(1+(auc-min)/(max-min))/2}
where auc is the uncorrected pAUC computed in the region defined by \code{partial.auc},
min is the value of the non-discriminant AUC (with an AUC of 0.5 or 50%)
in the region and max is the maximum possible AUC in the region. With this correction, the AUC
will be 0.5 if non discriminant and 1.0 if maximal, whatever the region
defined. This correction is fully compatible with \code{percent}.
Note that this correction is undefined for curves below the diagonal (auc < min). Attempting
to correct such an AUC will return \code{NA} with a warning.
}
\section{Smoothed ROC curves}{
There is no difference in the computation of the area under a smoothed
ROC curve, except for curves smoothed with \code{method="binomial"}. In this case
and only if a full AUC is requested, the classical binormal AUC formula is applied:
\deqn{auc=\phi\frac{a}{\sqrt{1 + b^2}}.}{pnorm(a/sqrt(1+b^2).}
If the ROC curve is smoothed with any other \code{method} or if a partial AUC
is requested, the empirical AUC described in the previous section is applied.
}
\section{Multi-class AUCs}{
With an object of class \dQuote{multiclass.roc}, a multi-class AUC is
computed as an average AUC as defined by Hand and Till (equation 7).
\deqn{auc=\frac{2}{c(c-1)}\sum{aucs}}{2/(count * (count -
1))*sum(aucs)}
with aucs all the pairwise roc curves.
}
\value{
The numeric AUC value, of class \code{c("auc", "numeric")} (or
\code{c("multiclass.auc", "numeric")} or \code{c("mv.multiclass.auc", "numeric")}
if a \dQuote{multiclass.roc} was supplied), in
fraction of the area or in percent if \code{percent=TRUE}, with the
following attributes:
\item{partial.auc}{if the AUC is full (FALSE) or partial (and in this
case the bounds), as defined in argument.}
\item{partial.auc.focus}{only for a partial AUC, if the bound specifies
the sensitivity or specificity, as defined in argument.}
\item{partial.auc.correct}{only for a partial AUC, was it corrected?
As defined in argument.}
\item{percent}{whether the AUC is given in percent or fraction.}
\item{roc}{the original ROC curve, as a \dQuote{\link{roc}},
\dQuote{\link{smooth.roc}} or \dQuote{\link{multiclass.roc}} object.}
}
\references{
Tom Fawcett (2006) ``An introduction to ROC analysis''. \emph{Pattern
Recognition Letters} \bold{27}, 861--874. DOI:
\doi{10.1016/j.patrec.2005.10.010}.
David J. Hand and Robert J. Till (2001). A Simple Generalisation of
the Area Under the ROC Curve for Multiple Class Classification
Problems. \emph{Machine Learning} \bold{45}(2), p. 171--186. DOI:
\doi{10.1023/A:1010920819831}.
Donna Katzman McClish (1989) ``Analyzing a Portion of the ROC
Curve''. \emph{Medical Decision Making} \bold{9}(3), 190--195. DOI:
\doi{10.1177/0272989X8900900307}.
Xavier Robin, Natacha Turck, Alexandre Hainard, \emph{et al.}
(2011) ``pROC: an open-source package for R and S+ to analyze and
compare ROC curves''. \emph{BMC Bioinformatics}, \bold{7}, 77.
DOI: \doi{10.1186/1471-2105-12-77}.
}
\seealso{
\code{\link{roc}}, \code{\link{ci.auc}}
}
\examples{
# Create a ROC curve:
data(aSAH)
roc.s100b <- roc(aSAH$outcome, aSAH$s100b)
# Get the full AUC
auc(roc.s100b)
# Get the partial AUC:
auc(roc.s100b, partial.auc=c(1, .8), partial.auc.focus="se", partial.auc.correct=TRUE)
}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{roc}
pROC/man/plot.ci.Rd 0000644 0001762 0000144 00000007663 15040443562 013506 0 ustar ligges users \encoding{UTF-8}
\name{plot.ci}
\alias{plot.ci}
\alias{plot.ci.thresholds}
\alias{plot.ci.se}
\alias{plot.ci.sp}
\alias{plot.ci.coords}
\title{
Plot confidence intervals
}
\description{
This function adds confidence intervals to a ROC curve plot, either as
bars or as a confidence shape.
}
\usage{
\S3method{plot}{ci.thresholds}(x, length=.01*ifelse(attr(x,
"roc")$percent, 100, 1), col=par("fg"), ...)
\S3method{plot}{ci.sp}(x, type=c("bars", "shape"), length=.01*ifelse(attr(x,
"roc")$percent, 100, 1), col=ifelse(type=="bars", par("fg"),
"gainsboro"), no.roc=FALSE, ...)
\S3method{plot}{ci.se}(x, type=c("bars", "shape"), length=.01*ifelse(attr(x,
"roc")$percent, 100, 1), col=ifelse(type=="bars", par("fg"),
"gainsboro"), no.roc=FALSE, ...)
\S3method{plot}{ci.coords}(x, type=c("bars", "shape"), length=NULL,
col=ifelse(type=="bars", par("fg"), "gainsboro"), ...)
}
\arguments{
\item{x}{a confidence interval object from the functions
\code{\link{ci.thresholds}}, \code{\link{ci.se}}, \code{\link{ci.sp}} or
\code{\link{ci.coords}}.
}
\item{type}{type of plot, \dQuote{bars} or \dQuote{shape}. Can be
shortened to \dQuote{b} or \dQuote{s}. \dQuote{shape} is only available for
\code{ci.se} and \code{ci.sp}, not for \code{ci.thresholds}.
}
\item{length}{the length (as plot coordinates) of the bar ticks. Only
if \code{type="bars"}.
}
\item{no.roc}{
if \code{FALSE}, the ROC line is re-added over the
shape. Otherwise if \code{TRUE}, only the shape is plotted. Ignored
if \code{type="bars"}
}
\item{col}{color of the bars or shape.}
\item{\dots}{further arguments for \code{\link{segments}} (if
\code{type="bars"}) or \code{\link{polygon}} (if
\code{type="shape"}).
}
}
\details{
This function adds confidence intervals to a ROC curve plot, either as
bars or as a confidence shape, depending on the state of the
\code{type} argument. The shape is plotted over the ROC curve, so that
the curve is re-plotted unless \code{no.roc=TRUE}.
Graphical functions are called with \link{suppressWarnings}.
}
\section{Warnings}{
With \code{type="shape"}, the warning \dQuote{Low definition shape} is
issued when the shape is defined by less than 15 confidence
intervals. In such a case, the shape is not well defined and the ROC
curve could pass outside the shape. To get a better shape, increase
the number of intervals, for example with:
\preformatted{plot(ci.sp(rocobj, sensitivities=seq(0, 1, .01)), type="shape")}
}
\value{
This function returns the confidence interval object invisibly.
}
\references{
Xavier Robin, Natacha Turck, Alexandre Hainard, \emph{et al.}
(2011) ``pROC: an open-source package for R and S+ to analyze and
compare ROC curves''. \emph{BMC Bioinformatics}, \bold{7}, 77.
DOI: \doi{10.1186/1471-2105-12-77}.
}
\seealso{
\code{\link{plot.roc}}, \code{\link{ci.thresholds}}, \code{\link{ci.sp}}, \code{\link{ci.se}}
}
\examples{
data(aSAH)
\dontrun{
# Start a ROC plot
rocobj <- plot.roc(aSAH$outcome, aSAH$s100b)
plot(rocobj)
# Thresholds
ci.thresolds.obj <- ci.thresholds(rocobj)
plot(ci.thresolds.obj)
# Specificities
plot(rocobj) # restart a new plot
ci.sp.obj <- ci.sp(rocobj, boot.n=500)
plot(ci.sp.obj)
# Sensitivities
plot(rocobj) # restart a new plot
ci.se.obj <- ci(rocobj, of="se", boot.n=500)
plot(ci.se.obj)
# Plotting a shape. We need more
ci.sp.obj <- ci.sp(rocobj, sensitivities=seq(0, 1, .01), boot.n=100)
plot(rocobj) # restart a new plot
plot(ci.sp.obj, type="shape", col="blue")
# Direct syntax (response, predictor):
plot.roc(aSAH$outcome, aSAH$s100b,
ci=TRUE, of="thresholds")
# CI of a PR curve
co <- coords(rocobj, x = "all", input="recall", ret=c("recall", "precision"))
ci <- ci.coords(rocobj, x = seq(0, 1, .1), input="recall", ret="precision")
plot(co, type="l", ylim = c(0, 1))
plot(ci, type="shape")
plot(ci, type="bars")
lines(co)
}
}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{aplot}
\keyword{hplot}
\keyword{roc}
pROC/man/ci.coords.Rd 0000644 0001762 0000144 00000017677 15040443562 014027 0 ustar ligges users \encoding{UTF-8}
\name{ci.coords}
\alias{ci.coords}
\alias{ci.coords.default}
\alias{ci.coords.formula}
\alias{ci.coords.roc}
\alias{ci.coords.smooth.roc}
\title{
Compute the confidence interval of arbitrary coordinates
}
\description{
This function computes the confidence interval (CI) of the coordinates
of a ROC curves with the \code{\link{coords}} function.
}
\usage{
# ci.coords(...)
\S3method{ci.coords}{roc}(roc, x,
input="threshold",
ret=c("threshold", "specificity", "sensitivity"),
best.method=c("youden", "closest.topleft"), best.weights=c(1, 0.5),
best.policy = c("stop", "omit", "random"),
conf.level=0.95, boot.n=2000,
boot.stratified=TRUE,
progress = NULL, ...)
\S3method{ci.coords}{formula}(formula, data, ...)
\S3method{ci.coords}{smooth.roc}(smooth.roc, x,
input=c("specificity", "sensitivity"), ret=c("specificity", "sensitivity"),
best.method=c("youden", "closest.topleft"), best.weights=c(1, 0.5),
best.policy = c("stop", "omit", "random"),
conf.level=0.95, boot.n=2000,
boot.stratified=TRUE,
progress = NULL, ...)
\S3method{ci.coords}{default}(response, predictor, ...)
}
\arguments{
\item{roc, smooth.roc}{a \dQuote{roc} object from the
\code{\link{roc}} function, or a \dQuote{smooth.roc} object from the
\code{\link[=smooth.roc]{smooth}} function.
}
\item{response, predictor}{arguments for the \code{\link{roc}} function.}
\item{formula, data}{a formula (and possibly a data object) of type
response~predictor for the \code{\link{roc}} function.
}
\item{x, input, ret, best.method, best.weights}{Arguments passed to \code{\link{coords}}.
See there for more details. The only difference is on the \code{x} argument which cannot be
\dQuote{all} or \dQuote{local maximas}.
}
\item{best.policy}{The policy follow when multiple \dQuote{best} thresholds are returned by \code{\link{coords}}.
\dQuote{stop} will abort the processing with \code{\link{stop}} (default),
\dQuote{omit} will ignore the sample (as in \code{\link{NA}})
and \dQuote{random} will select one of the threshold randomly.
}
\item{conf.level}{the width of the confidence interval as [0,1], never
in percent. Default: 0.95, resulting in a 95\% CI.
}
\item{boot.n}{the number of bootstrap replicates. Default: 2000.}
\item{boot.stratified}{should the bootstrap be stratified (default, same number
of cases/controls in each replicate than in the original sample) or
not.
}
\item{progress}{DEPRECATED. A value other than \code{NULL} will produce
a warning. The argument will be removed in a future version.}
\item{\dots}{further arguments passed to or from other methods,
especially arguments for \code{\link{roc}} and \code{ci.coords.roc}
when calling \code{ci.coords.default} or \code{ci.coords.formula}.
Arguments for \code{\link{txtProgressBar}} (only
\code{char} and \code{style}) if applicable.
}
}
\details{
\code{ci.coords.formula} and \code{ci.coords.default} are convenience methods
that build the ROC curve (with the \code{\link{roc}} function) before
calling \code{ci.coords.roc}. You can pass them arguments for both
\code{\link{roc}} and \code{ci.coords.roc}. Simply use \code{ci.coords}
that will dispatch to the correct method.
This function creates \code{boot.n} bootstrap replicate of the ROC
curve, and evaluates the coordinates specified by the \code{x}, \code{input},
\code{ret}, \code{best.method} and \code{best.weights} arguments. Then it computes the
confidence interval as the percentiles given by \code{conf.level}.
When \code{x="best"}, the best threshold is determined at each bootstrap
iteration, effectively assessing the confidence interval of choice of the "best"
threshold itself. This differs from the behavior of \code{\link{ci.thresholds}},
where the "best" threshold is assessed on the given ROC curve before
resampling.
For more details about the bootstrap, see the Bootstrap section in
\link[=pROC-package]{this package's documentation}.
}
\section{Warnings}{
If \code{boot.stratified=FALSE} and the sample has a large imbalance between
cases and controls, it could happen that one or more of the replicates
contains no case or control observation, producing a \code{NA} area.
The warning \dQuote{NA value(s) produced during bootstrap were ignored.}
will be issued and the observation will be ignored. If you have a large
imbalance in your sample, it could be safer to keep
\code{boot.stratified=TRUE}.
This warning will also be displayed if you chose \code{best.policy = "omit"}
and a ROC curve with multiple \dQuote{best} threshold was generated
during at least one of the replicates.
}
\value{
\bold{Note:} changed in version 1.16.
A list of the same length as \code{ret} and named as \code{ret}, and of
class \dQuote{ci.thresholds}, \dQuote{ci} and \dQuote{list} (in this order).
Each element of the list is a matrix of the confidence intervals with
rows given by \code{x} and with 3 columns, the lower bound of the CI,
the median, and the upper bound of the CI.
Additionally, the list has the following attributes:
\item{conf.level}{the width of the CI, in fraction.}
\item{boot.n}{the number of bootstrap replicates.}
\item{boot.stratified}{whether or not the bootstrapping was stratified.}
\item{input}{the input coordinate, as given in argument.}
\item{x}{the coordinates used to calculate the CI, as given in argument.}
\item{ret}{the return values, as given in argument or substituted by
\code{link{coords}}.}
\item{roc}{the object of class \dQuote{\link{roc}} that was used to
compute the CI.
}
}
\references{
James Carpenter and John Bithell (2000) ``Bootstrap condence intervals:
when, which, what? A practical guide for medical statisticians''.
\emph{Statistics in Medicine} \bold{19}, 1141--1164.
DOI: \doi{10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F}.
Tom Fawcett (2006) ``An introduction to ROC analysis''. \emph{Pattern
Recognition Letters} \bold{27}, 861--874. DOI:
\doi{10.1016/j.patrec.2005.10.010}.
Hadley Wickham (2011) ``The Split-Apply-Combine Strategy for Data Analysis''. \emph{Journal of Statistical Software}, \bold{40}, 1--29.
URL: \doi{10.18637/jss.v040.i01}.
}
\seealso{
\code{\link{roc}},
\code{\link{coords}},
\code{\link{ci}}
}
\examples{
# Create a ROC curve:
data(aSAH)
roc1 <- roc(aSAH$outcome, aSAH$s100b)
## Basic example ##
\dontrun{
ci.coords(roc1, x="best", input = "threshold",
ret=c("specificity", "ppv", "tp"))
## More options ##
ci.coords(roc1, x=0.9, input = "sensitivity", ret="specificity")
ci.coords(roc1, x=0.9, input = "sensitivity", ret=c("specificity", "ppv", "tp"))
ci.coords(roc1, x=c(0.1, 0.5, 0.9), input = "sensitivity", ret="specificity")
ci.coords(roc1, x=c(0.1, 0.5, 0.9), input = "sensitivity", ret=c("specificity", "ppv", "tp"))
# Return everything we can:
rets <- c("threshold", "specificity", "sensitivity", "accuracy", "tn", "tp", "fn", "fp", "npv",
"ppv", "1-specificity", "1-sensitivity", "1-accuracy", "1-npv", "1-ppv")
ci.coords(roc1, x="best", input = "threshold", ret=rets)}\dontshow{
ci.coords(roc1, x=0.9, input = "sensitivity", ret="specificity", boot.n=10)
ci.coords(roc1, x=0.9, input = "sensitivity", ret=c("specificity", "ppv", "tp"), boot.n=10)
ci.coords(roc1, x=c(0.1, 0.5, 0.9), input = "sensitivity", ret="specificity", boot.n=10)
ci.coords(roc1, x=c(0.1, 0.5, 0.9), input = "sensitivity", ret=c("specificity", "ppv", "tp"), boot.n=10)
rets <- c("threshold", "specificity", "sensitivity", "accuracy", "tn", "tp", "fn", "fp", "npv",
"ppv", "1-specificity", "1-sensitivity", "1-accuracy", "1-npv", "1-ppv")
ci.coords(roc1, x="best", input = "threshold", ret=rets, boot.n=10)}
## On smoothed ROC curves with bootstrap ##
\dontrun{
ci.coords(smooth(roc1), x=0.9, input = "sensitivity", ret=c("specificity", "ppv", "tp"))}\dontshow{
ci.coords(smooth(roc1), x=0.9, input = "sensitivity", ret=c("specificity", "ppv", "tp"), boot.n = 10)}
}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{roc}
pROC/man/roc.test.Rd 0000644 0001762 0000144 00000047557 15040443562 013705 0 ustar ligges users \encoding{UTF-8}
\name{roc.test}
\alias{roc.test}
\alias{roc.test.default}
\alias{roc.test.roc}
\alias{roc.test.formula}
\alias{roc.test.auc}
\alias{roc.test.smooth.roc}
\title{
Compare two ROC curves
}
\description{
This function compares two correlated (or paired) or uncorrelated (unpaired)
ROC curves. Delong and bootstrap methods test for a difference in the
(partial) AUC of the ROC curves. The Venkatraman method tests if the two
curves are perfectly superposed. The sensitivity and specificity methods
test if the sensitivity (respectively specificity) of the ROC curves are
different at the given level of specificity (respectively sensitivity).
Several syntaxes are available: two object of class roc (which can be AUC
or smoothed ROC), or either three vectors (response, predictor1, predictor2)
or a response vector and a matrix or data.frame with two columns
(predictors).
}
\usage{
# roc.test(...)
\S3method{roc.test}{roc}(roc1, roc2, method=c("delong", "bootstrap",
"venkatraman", "sensitivity", "specificity"), sensitivity = NULL,
specificity = NULL, alternative = c("two.sided", "less", "greater"),
paired=NULL, reuse.auc=TRUE, boot.n=2000, boot.stratified=TRUE,
ties.method="first", progress = NULL,
parallel=FALSE, conf.level=0.95, ...)
\S3method{roc.test}{auc}(roc1, roc2, ...)
\S3method{roc.test}{smooth.roc}(roc1, roc2, ...)
\S3method{roc.test}{formula}(formula, data, ...)
\S3method{roc.test}{default}(response, predictor1, predictor2=NULL,
na.rm=TRUE, method=NULL, ...)
}
\arguments{
\item{roc1, roc2}{the two ROC curves to compare. Either
\dQuote{\link{roc}}, \dQuote{\link{auc}} or
\dQuote{\link{smooth.roc}} objects (types can be mixed).
}
\item{response}{a vector or factor, as for the \link{roc} function.}
\item{predictor1}{a numeric or ordered vector as for the \link{roc} function, or
a matrix or data.frame with predictors two colums.}
\item{predictor2}{only if predictor1 was a vector, the second
predictor as a numeric vector.
}
\item{formula}{a formula of the type response~predictor1+predictor2.
Additional arguments \code{data}, \code{subset} and \code{na.action}
are supported, see \code{\link{model.frame}} for more details.
}
\item{data}{a matrix or data.frame containing the variables in the
formula. See \code{\link{model.frame}} for more details.}
\item{na.rm}{if \code{TRUE}, the observations with \code{NA} values
will be removed.
}
\item{method}{the method to use, either \dQuote{delong},
\dQuote{bootstrap} or \dQuote{venkatraman}. The first letter is sufficient. If omitted, the
appropriate method is selected as explained in details.
}
\item{sensitivity, specificity}{if \code{method="sensitivity"} or
\code{method="specificity"}, the respective level where the test
must be assessed as a numeric of length 1. }
\item{alternative}{specifies the alternative hypothesis. Either of
\dQuote{two.sided}, \dQuote{less} or \dQuote{greater}. The first letter is
sufficient. Default: \dQuote{two.sided}. Only \dQuote{two.sided} is available
with \code{method="venkatraman"}.
}
\item{paired}{a logical indicating whether you want a paired roc.test.
If \code{NULL}, the paired status will be auto-detected by \code{\link{are.paired}}.
If \code{TRUE} but the paired status cannot be assessed by \code{\link{are.paired}}
will produce an error.
}
\item{reuse.auc}{if \code{TRUE} (default) and the \dQuote{roc} objects
contain an \dQuote{auc} field, re-use these specifications for the
test. See the \emph{AUC specification} section for more details.
}
\item{boot.n}{for \code{method="bootstrap"} and \code{method="venkatraman"} only: the number of
bootstrap replicates or permutations. Default: \var{2000}.
}
\item{boot.stratified}{for \code{method="bootstrap"} only:
should the bootstrap be stratified (same number
of cases/controls in each replicate than in the original sample) or
not. Ignored with \code{method="venkatraman"}. Default: \var{TRUE}.
}
\item{ties.method}{for \code{method="venkatraman"} only: argument for
\code{\link{rank}} specifying how ties are handled. Defaults to
\dQuote{first} as described in the paper.
}
\item{progress}{DEPRECATED. A value other than \code{NULL} will produce
a warning. The argument will be removed in a future version.}
\item{parallel}{DEPRECATED. A value other than \code{FALSE} will produce a warning.
The argument will be removed in a future version.}
\item{conf.level}{a numeric scalar between 0 and 1 (non-inclusive) which
species the confidence level to use for any calculated CI's.}
\item{\dots}{further arguments passed to or from other methods,
especially arguments for \code{\link{roc}} and \code{roc.test.roc}
when calling \code{roc.test.default} or \code{roc.test.formula}.
Arguments for \code{\link{auc}},
and \code{\link{txtProgressBar}} (only \code{char} and \code{style})
if applicable.
}
}
\details{
This function compares two ROC curves. It is typically called with the two \link{roc} objects to
compare. \code{roc.test.default} is provided as a convenience
method and creates two \link{roc} objects before calling
\code{roc.test.roc}.
Three methods are available: \dQuote{delong}, \dQuote{bootstrap} and \dQuote{venkatraman} (see
\dQuote{Computational details} section below). \dQuote{delong} and
\dQuote{bootstrap} are tests over the AUC whereas \dQuote{venkatraman}
compares the the ROC curves themselves.
Default is to use \dQuote{delong} method except for comparison of partial AUC, smoothed
curves and curves with different \code{direction}, where \code{bootstrap}
is used. Using \dQuote{delong} for partial AUC and smoothed ROCs is not
supported in pROC and result in an error.
It is spurious to use \dQuote{delong} for \code{\link{roc}} with different
\code{direction} (a warning is issued but the spurious comparison is
enforced). \dQuote{venkatraman}'s test cannot be employed to compare smoothed
ROC curves, or curves with partial AUC specifications. In addition,
and comparison of ROC curves with different
\code{direction} should be used with care (a warning is produced as well).
If \code{alternative="two.sided"}, a two-sided test for difference in AUC is performed. If
\code{alternative="less"}, the alternative is that the AUC of roc1 is
smaller than the AUC of roc2. For \code{method="venkatraman"}, only
\dQuote{two.sided} test is available.
If the \code{paired} argument is not provided, the \code{\link{are.paired}} function is
employed to detect the paired status of the ROC curves. It will test if the original \code{response} is
identical between the two ROC curves (this is always the case if the call is made with
\code{roc.test.default}). This detection is unlikely to raise false positives, but
this possibility cannot be excluded entierly. It would require equal sample sizes
and \code{response} values and order in both ROC curves. If it happens to you, use \code{paired=FALSE}.
If you know the ROC curves are paired you can pass \code{paired=TRUE}. However this is
useless as it will be tested anyway.
For \link[=smooth.roc]{smoothed ROC curves}, smoothing is performed again at each
bootstrap replicate with the parameters originally provided.
If a density smoothing was performed with user-provided
\code{density.cases} or \code{density.controls} the bootstrap cannot
be performed and an error is issued.
}
\section{AUC specification}{
The comparison of the AUC of the ROC curves needs a specification of the
AUC. The specification is defined by:
\enumerate{
\item the \dQuote{auc} field in the \dQuote{\link{roc}} objects if
\code{reuse.auc} is set to \code{TRUE} (default)
\item passing the specification to \code{\link{auc}} with \dots
(arguments \code{partial.auc}, \code{partial.auc.correct} and
\code{partial.auc.focus}). In this case, you must ensure either that
the \code{\link{roc}} object do not contain an \code{auc} field (if
you called \code{\link{roc}} with \code{auc=FALSE}), or set
\code{reuse.auc=FALSE}.
}
If \code{reuse.auc=FALSE} the \code{\link{auc}} function will always
be called with \code{\dots} to determine the specification, even if
the \dQuote{\link{roc}} objects do contain an \code{auc} field.
As well if the \dQuote{\link{roc}} objects do not contain an \code{auc}
field, the \code{\link{auc}} function will always be called with
\code{\dots} to determine the specification.
The AUC specification is ignored in the Venkatraman test.
Warning: if the roc object passed to roc.test contains an \code{auc}
field and \code{reuse.auc=TRUE}, \link{auc} is not called and
arguments such as \code{partial.auc} are silently ignored.
}
\section{Computation details}{
With \code{method="bootstrap"}, the processing is done as follow:
\enumerate{
\item \code{boot.n} bootstrap replicates are drawn from the
data. If \code{boot.stratified} is \var{TRUE}, each replicate contains
exactly the same number of controls and cases than the original
sample, otherwise if \var{FALSE} the numbers can vary.
\item for each bootstrap replicate, the AUC of the two ROC curves
are computed and the difference is stored.
\item The following formula is used:
\deqn{D=\frac{AUC1-AUC2}{s}}{D=(AUC1-AUC2)/s}
where s is the standard deviation of
the bootstrap differences and AUC1 and AUC2 the AUC of the two
(original) ROC curves.
\item \var{D} is then compared to the normal distribution,
according to the value of \code{alternative}.
}
See also the Bootstrap section in
\link[=pROC-package]{this package's documentation}.
With \code{method="delong"}, the processing is done as described in
DeLong \emph{et al.} (1988) for paired ROC curves, using the algorithm
of Sun and Xu (2014). Only comparison of
two ROC curves is implemented. The method has been extended for
unpaired ROC curves where the p-value is computed with an unpaired
t-test with unequal sample size and unequal variance, with
\deqn{
D=\frac{V^r(\theta^r) - V^s(\theta^s) }{ \sqrt{S^r + S^s}}
}{
D=(V^r(\theta^r) - V^s(\theta^s)) / sqrt(S^r + S^s)
}
With \code{method="venkatraman"}, the processing is done as described
in Venkatraman and Begg (1996) (for paired ROC curves) and Venkatraman
(2000) (for unpaired ROC curves) with \code{boot.n} permutation of
sample ranks (with ties breaking). For consistency reasons, the same argument \code{boot.n} as
in bootstrap defines the number of permutations to execute,
even though no bootstrap is performed.
For \code{method="specificity"}, the test assesses if the sensitivity of
the ROC curves are different at the level of specificity given by the
\code{specificity} argument, which must be a numeric of length 1. Bootstrap is employed as with \code{method="bootstrap"}
and \code{boot.n} and \code{boot.stratified} are available. This is
identical to the test proposed by Pepe \emph{et al.} (2009).
The \code{method="sensitivity"} is very similar, but assesses if the specificity of
the ROC curves are different at the level of sensitivity given by the
\code{sensitivity} argument.
}
\section{Warnings}{
If \dQuote{auc} specifications are different in both roc objects, the
warning \dQuote{Different AUC specifications in the ROC
curves. Enforcing the inconsistency, but unexpected results may be
produced.} is issued. Unexpected results may be produced.
If one or both ROC curves are \dQuote{smooth.roc} objects with
different smoothing specifications, the warning
\dQuote{Different smoothing parameters in the ROC curves. Enforcing
the inconsistency, but unexpected results may be produced.} is issued.
This warning can be benign, especially if ROC curves were generated
with \code{roc(\ldots, smooth=TRUE)} with different arguments to other
functions (such as plot), or if you really want to compare two ROC
curves smoothed differently.
If \code{method="venkatraman"}, and \code{alternative} is
\dQuote{less} or \dQuote{greater}, the warning \dQuote{Only two-sided
tests are available for Venkatraman. Performing two-sided test instead.}
is produced and a two tailed test is performed.
Both DeLong and Venkatraman's test ignores the direction of the ROC curve so that if two
ROC curves have a different differ in the value of
\code{direction}, the warning \dQuote{(DeLong|Venkatraman)'s test should not be
applied to ROC curves with different directions.} is
printed. However, the spurious test is enforced.
If \code{boot.stratified=FALSE} and the sample has a large imbalance between
cases and controls, it could happen that one or more of the replicates
contains no case or control observation, or that there are not enough
points for smoothing, producing a \code{NA} area.
The warning \dQuote{NA value(s) produced during bootstrap were ignored.}
will be issued and the observation will be ignored. If you have a large
imbalance in your sample, it could be safer to keep
\code{boot.stratified=TRUE}.
When both ROC curves have an \code{\link{auc}} of 1 (or 100\%), their variances and covariance will always be null,
and therefore the p-value will always be 1. This is true for both \dQuote{delong}, \dQuote{bootstrap} and
\dQuote{venkatraman} methods. This result is misleading, as the variances and covariance are of course not null.
A \code{\link{warning}} will be displayed to inform of this condition, and of the misleading output.
}
\section{Errors}{
An error will also occur if you give a \code{predictor2} when
\code{predictor1} is a \code{\link{matrix}} or a
\code{\link{data.frame}}, if \code{predictor1} has more than two
columns, or if you do not give a \code{predictor2} when
\code{predictor1} is a vector.
If \code{density.cases} and \code{density.controls} were provided
for smoothing, the error \dQuote{Cannot compute the statistic on ROC
curves smoothed with density.controls and density.cases.} is
issued.
If \code{method="venkatraman"} and one of the ROC curves is smoothed,
the error \dQuote{Using Venkatraman's test for smoothed ROCs is not
supported.} is produced.
With \code{method="specificity"}, the error \dQuote{Argument
'specificity' must be numeric of length 1 for a specificity test.}
is given unless the specificity argument is specified as a numeric of
length 1. The \dQuote{Argument 'sensitivity' must be numeric of length
1 for a sensitivity test.} message is given for
\code{method="sensitivity"} under similar conditions.
}
\value{
A list of class "htest" with following content:
\item{p.value}{the p-value of the test.}
\item{statistic}{the value of the Z (\code{method="delong"}) or D
(\code{method="bootstrap"}) statistics.
}
\item{conf.int}{the confidence interval of the test (currently only returned for the paired DeLong's test). Has an attribute \code{conf.level} specifiying the level of the test.}
\item{alternative}{the alternative hypothesis.}
\item{method}{the character string \dQuote{DeLong's test for two
correlated ROC curves} (if \code{method="delong"}) or
\dQuote{Bootstrap test for two correlated ROC curves} (if
\code{method="bootstrap"}).
}
\item{null.value}{the expected value of the statistic under the null
hypothesis, that is 0.}
\item{estimate}{the AUC in the two ROC curves.}
\item{data.name}{the names of the data that was used.}
\item{parameter}{for \code{method="bootstrap"} only: the values of the
\code{boot.n} and \code{boot.stratified} arguments.
}
}
\section{Acknowledgements}{
We would like to thank E. S. Venkatraman and Colin B. Begg for their
support in the implementation of their test.
}
\references{
Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson
(1988) ``Comparing the areas under two or more correlated receiver
operating characteristic curves: a nonparametric
approach''. \emph{Biometrics} \bold{44}, 837--845.
James A. Hanley and Barbara J. McNeil (1982) ``The meaning and use of
the area under a receiver operating characteristic (ROC)
curve''. \emph{Radiology} \bold{143}, 29--36.
Margaret Pepe, Gary Longton and Holly Janes (2009) ``Estimation and
Comparison of Receiver Operating Characteristic Curves''. \emph{The
Stata journal} \bold{9}, 1.
Xavier Robin, Natacha Turck, Jean-Charles Sanchez and Markus Müller
(2009) ``Combination of protein biomarkers''. \emph{useR! 2009}, Rennes.
\url{https://www.r-project.org/nosvn/conferences/useR-2009/abstracts/user_author.html}
Xavier Robin, Natacha Turck, Alexandre Hainard, \emph{et al.}
(2011) ``pROC: an open-source package for R and S+ to analyze and
compare ROC curves''. \emph{BMC Bioinformatics}, \bold{7}, 77.
DOI: \doi{10.1186/1471-2105-12-77}.
Xu Sun and Weichao Xu (2014) ``Fast Implementation of DeLongs Algorithm for Comparing
the Areas Under Correlated Receiver Operating Characteristic Curves''. \emph{IEEE Signal
Processing Letters}, \bold{21}, 1389--1393.
DOI: \doi{10.1109/LSP.2014.2337313}.
E. S. Venkatraman and Colin B. Begg (1996) ``A distribution-free
procedure for comparing receiver operating characteristic curves from
a paired experiment''. \emph{Biometrika} \bold{83}, 835--848.
DOI: \doi{10.1093/biomet/83.4.835}.
E. S. Venkatraman (2000) ``A Permutation Test to Compare Receiver
Operating Characteristic Curves''. \emph{Biometrics} \bold{56},
1134--1138. DOI: \doi{10.1111/j.0006-341X.2000.01134.x}.
Hadley Wickham (2011) ``The Split-Apply-Combine Strategy for Data Analysis''. \emph{Journal of Statistical Software}, \bold{40}, 1--29.
URL: \doi{10.18637/jss.v040.i01}.
}
\seealso{
\code{\link{roc}}, \code{\link{power.roc.test}}
}
\examples{
data(aSAH)
# Basic example with 2 roc objects
roc1 <- roc(aSAH$outcome, aSAH$s100b)
roc2 <- roc(aSAH$outcome, aSAH$wfns)
roc.test(roc1, roc2)
\dontrun{
# The latter used Delong's test. To use bootstrap test:
roc.test(roc1, roc2, method="bootstrap")
# Increase boot.n for a more precise p-value:
roc.test(roc1, roc2, method="bootstrap", boot.n=10000)
}
# Alternative syntaxes
roc.test(aSAH$outcome, aSAH$s100b, aSAH$wfns)
roc.test(aSAH$outcome, data.frame(aSAH$s100b, aSAH$wfns))
# If we had a good a priori reason to think that wfns gives a
# better classification than s100b (in other words, AUC of roc1
# should be lower than AUC of roc2):
roc.test(roc1, roc2, alternative="less")
\dontrun{
# Comparison can be done on smoothed ROCs
# Smoothing is re-done at each iteration, and execution is slow
roc.test(smooth(roc1), smooth(roc2))
# or:
roc.test(aSAH$outcome, aSAH$s100b, aSAH$wfns, smooth=TRUE, boot.n=100)
}
# or from an AUC (no smoothing)
roc.test(auc(roc1), roc2)
\dontrun{
# Comparison of partial AUC:
roc3 <- roc(aSAH$outcome, aSAH$s100b, partial.auc=c(1, 0.8), partial.auc.focus="se")
roc4 <- roc(aSAH$outcome, aSAH$wfns, partial.auc=c(1, 0.8), partial.auc.focus="se")
roc.test(roc3, roc4)
# This is strictly equivalent to:
roc.test(roc3, roc4, method="bootstrap")
# Alternatively, we could re-use roc1 and roc2 to get the same result:
roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(1, 0.8), partial.auc.focus="se")
# Comparison on specificity and sensitivity
roc.test(roc1, roc2, method="specificity", specificity=0.9)
roc.test(roc1, roc2, method="sensitivity", sensitivity=0.9)
}
# Spurious use of DeLong's test with different direction:
roc5 <- roc(aSAH$outcome, aSAH$s100b, direction="<")
roc6 <- roc(aSAH$outcome, aSAH$s100b, direction=">")
roc.test(roc5, roc6, method="delong")
\dontrun{
# Comparisons of the ROC curves
roc.test(roc1, roc2, method="venkatraman")
}
# Unpaired tests
roc7 <- roc(aSAH$outcome, aSAH$s100b)
# artificially create an roc8 unpaired with roc7
roc8 <- roc(aSAH$outcome[1:100], aSAH$s100b[1:100])
\dontrun{
roc.test(roc7, roc8, paired=FALSE, method="delong")
roc.test(roc7, roc8, paired=FALSE, method="bootstrap")
roc.test(roc7, roc8, paired=FALSE, method="venkatraman")
roc.test(roc7, roc8, paired=FALSE, method="specificity", specificity=0.9)
}
}
\keyword{multivariate}
\keyword{nonparametric}
\keyword{utilities}
\keyword{htest}
\keyword{roc}
pROC/man/coords.Rd 0000644 0001762 0000144 00000033156 15040443562 013423 0 ustar ligges users \encoding{UTF-8}
\name{coords}
\alias{coords}
\alias{coords.auc}
\alias{coords.roc}
\alias{coords.smooth.roc}
\title{
Coordinates of a ROC curve
}
\description{
This function returns the coordinates of the ROC curve at one
or several specified point(s).
}
\usage{
coords(...)
\S3method{coords}{auc}(auc, ...)
\S3method{coords}{roc}(roc, x, input="threshold", ret=c("threshold",
"specificity", "sensitivity"), ignore.partial.auc=FALSE,
as.list=FALSE, drop=TRUE, best.method=c("youden", "closest.topleft"),
best.weights=c(1, 0.5), transpose = FALSE, as.matrix=FALSE, ...)
\S3method{coords}{smooth.roc}(smooth.roc, x, input, ret=c("specificity",
"sensitivity"), ignore.partial.auc=FALSE, as.list=FALSE, drop=TRUE,
best.method=c("youden", "closest.topleft"), best.weights=c(1, 0.5),
transpose = FALSE, as.matrix=FALSE, ...)
}
\arguments{
\item{auc, roc, smooth.roc}{a \dQuote{roc} object from the
\code{\link{roc}} function, or a \dQuote{smooth.roc} object from the
\code{\link[=smooth.roc]{smooth}} function, or an \dQuote{auc} object
from the \code{\link{auc}} function.
}
\item{x}{
the coordinates to look for. Numeric (if so, their meaning is
defined by the \code{input} argument) or one of \dQuote{all} (all
the points of the ROC curve), \dQuote{local maximas} (the local
maximas of the ROC curve) or \dQuote{best} (see \code{best.method}
argument). If missing or \code{NULL}, defaults to \dQuote{all}.
}
\item{input}{
If \code{x} is numeric, the kind of input coordinate (x).
Typically one of \dQuote{threshold}, \dQuote{specificity} or
\dQuote{sensitivity}, but can be any of the monotone coordinate available,
see the \dQuote{Valid input} column under \dQuote{Available coordinates}.
Can be shortened like \code{ret}. Defaults to \dQuote{threshold}. Note
that \dQuote{threshold} is not allowed in \code{coords.smooth.roc} and that
the argument is ignored when \code{x} is a character.
}
\item{ret}{The coordinates to return. See \dQuote{Available coordinates}
section below. Alternatively, the single value \dQuote{all} can be used to return
every coordinate available.
}
\item{ignore.partial.auc}{If the \code{roc} object contains a partial AUC
specification, it will be ignored.
}
\item{as.list}{DEPRECATED. If the returned object must be a list.
Will be removed in a future version.
}
\item{drop}{DEPRECATED. If \code{TRUE} the result is coerced to the lowest
possible dimension, as per \link{Extract}. By default only drops
if \code{transpose = TRUE} and either \code{ret} or \code{x} is
of length 1.
}
\item{best.method}{if \code{x="best"}, the method to determine the
best threshold. Defaults to "youden". See details in the
\sQuote{Best thresholds} section.
}
\item{best.weights}{if \code{x="best"}, the weights to determine the
best threshold. See details in the \sQuote{Best thresholds} section.
}
\item{transpose}{DEPRECATED. Whether
to return the thresholds in columns (\code{TRUE}) or rows (\code{FALSE}).
Since pROC 1.16 the default value is \code{FALSE}.
See \link{coords_transpose} for more details the change.
}
\item{as.matrix}{DEPRECATED. If \code{transpose} is \code{FALSE}, whether to return
a \code{\link{matrix}} (\code{TRUE}) or a \code{\link{data.frame}}
(\code{FALSE}, the default). A \code{data.frame} is more convenient
and flexible to use, but incurs a slight speed penalty. Consider
setting this argument to \code{TRUE} if you are calling the function
repeatedly.
}
\item{\dots}{further arguments passed from other methods. Ignored.}
}
\details{
This function takes a \dQuote{roc} or \dQuote{smooth.roc} object as
first argument, on which the coordinates will be determined. The
coordinates are defined by the \code{x} and \code{input}
arguments. \dQuote{threshold} coordinates cannot be determined in a
smoothed ROC.
If \code{input="threshold"}, the coordinates for the threshold
are reported, even if the exact threshold do not define the ROC
curve. The following convenience characters are allowed: \dQuote{all},
\dQuote{local maximas} and \dQuote{best}. They will return all the
thresholds, only the thresholds defining local maximas (upper angles of the
ROC curve), or only the threshold(s) corresponding to the best sum of
sensitivity + specificity respectively. Note that \dQuote{best} can
return more than one threshold. If \code{x} is a character, and
\code{ignore.partial.auc=TRUE}, the coordinates are limited to the
thresholds within the partial AUC if it has been defined, and not
necessarily to the whole curve.
For \code{input="specificity"} and \code{input="sensitivity"},
the function checks if the specificity or sensitivity is one of the
points of the ROC curve (in \code{roc$sensitivities} or
\code{roc$specificities}). More than one point may match (in
\emph{step} curves), then only the upper-left-most point coordinates
are returned. Otherwise,
the specificity and specificity of the point is interpolated and
\code{NA} is returned as threshold.
The coords function in this package is a generic, but it might be
superseded by functions in other packages such as
\pkg{colorspace} or \pkg{spatstat} if they are loaded after
\pkg{pROC}. In this case, call the \code{pROC::coords} explicitly.
\subsection{Best thresholds}{
If \code{x="best"}, the \code{best.method} argument controls how the
optimal threshold is determined.
\describe{
\item{\dQuote{youden}}{
Youden's J statistic (Youden, 1950) is employed (default). The optimal
cut-off is the threshold that maximizes the distance to the identity
(diagonal) line. Can be shortened to \dQuote{y}.
The optimality criterion is:
\deqn{max(sensitivities + specificities)}{max(sensitivities + specificities)}
}
\item{\dQuote{closest.topleft}}{
The optimal threshold is the point closest to the top-left part of
the plot with perfect sensitivity or specificity. Can be shortened
to \dQuote{c} or \dQuote{t}.
The optimality criterion is:
\deqn{min((1 - sensitivities)^2 + (1- specificities)^2)}{min((1 - sensitivities)^2 + (1- specificities)^2)}
}
}
In addition, weights can be supplied if false positive and false
negative predictions are not equivalent: a numeric vector of length 2
to the \code{best.weights} argument. The elements define
\enumerate{
\item the relative cost of of a false negative classification (as compared with a false positive classification)
\item the prevalence, or the proportion of cases in the population (\eqn{\frac{n_{cases}}{n_{controls}+n_{cases}}}{n.cases/(n.controls+n.cases)}).
}
The optimality criteria are modified as proposed by Perkins and Schisterman:
\describe{
\item{\dQuote{youden}}{
\deqn{max(sensitivities + r * specificities)}{max(sensitivities + r \times specificities)}
}
\item{\dQuote{closest.topleft}}{
\deqn{min((1 - sensitivities)^2 + r * (1- specificities)^2)}{min((1 - sensitivities)^2 + r \times (1- specificities)^2)}
}
}
with
\deqn{r = \frac{1 - prevalence}{cost * prevalence}}{r = (1 - prevalence) / (cost * prevalence)}
By default, prevalence is 0.5 and cost is 1 so that no weight is
applied in effect.
Note that several thresholds might be equally optimal.
}
\subsection{Available coordinates}{
The following table lists the coordinates that are available in the \code{ret}
and \code{input} arguments.
\tabular{rllll}{
Value \tab Description \tab Formula \tab Synonyms \tab Valid input \cr
\code{threshold} \tab The threshold value \tab - \tab - \tab Yes \cr
\code{tn} \tab True negative count \tab - \tab - \tab Yes \cr
\code{tp} \tab True positive count \tab - \tab - \tab Yes \cr
\code{fn} \tab False negative count \tab - \tab - \tab Yes \cr
\code{fp} \tab False positive count \tab - \tab - \tab Yes \cr
\code{specificity} \tab Specificity \tab tn / (tn + fp) \tab tnr \tab Yes \cr
\code{sensitivity} \tab Sensitivity \tab tp / (tp + fn) \tab recall, tpr \tab Yes \cr
\code{accuracy} \tab Accuracy \tab (tp + tn) / N \tab - \tab No \cr
\code{npv} \tab Negative Predictive Value \tab tn / (tn + fn) \tab - \tab No \cr
\code{ppv} \tab Positive Predictive Value \tab tp / (tp + fp) \tab precision \tab No \cr
\code{precision} \tab Precision \tab tp / (tp + fp) \tab ppv \tab No \cr
\code{recall} \tab Recall \tab tp / (tp + fn) \tab sensitivity, tpr \tab Yes \cr
\code{tpr} \tab True Positive Rate \tab tp / (tp + fn) \tab sensitivity, recall \tab Yes \cr
\code{fpr} \tab False Positive Rate \tab fp / (tn + fp) \tab 1-specificity \tab Yes \cr
\code{tnr} \tab True Negative Rate \tab tn / (tn + fp) \tab specificity \tab Yes \cr
\code{fnr} \tab False Negative Rate \tab fn / (tp + fn) \tab 1-sensitivity \tab Yes \cr
\code{fdr} \tab False Discovery Rate \tab fp / (tp + fp) \tab 1-ppv \tab No \cr
\code{lr_pos} \tab Positive Likelihood Ratio \tab se / (1 - sp) \tab - \tab No \cr
\code{lr_neg} \tab Negative Likelihood Ratio \tab (1 - se) / (sp) \tab - \tab No \cr
\code{youden} \tab Youden Index \tab
se + r * sp
\tab - \tab No \cr
\code{closest.topleft} \tab Distance to the top left corner of the ROC space \tab
- ((1 - se)^2 + r * (1 - sp)^2)
\tab - \tab No \cr
}
The value \dQuote{threshold} is not allowed in \code{coords.smooth.roc}.
Values can be shortenend (for example to \dQuote{thr}, \dQuote{sens} and \dQuote{spec}, or even to
\dQuote{se}, \dQuote{sp} or \dQuote{1-np}). In addition, some values can be prefixed with
\code{1-} to get their complement:
\code{1-specificity}, \code{1-sensitivity}, \code{1-accuracy}, \code{1-npv}, \code{1-ppv}.
The values \code{npe} and \code{ppe} are automatically replaced with
\code{1-npv} and \code{1-ppv}, respectively (and will therefore not appear
as is in the output, but as \code{1-npv} and \code{1-ppv} instead).
These must be used verbatim in ROC curves with \code{percent=TRUE}
(ie. \dQuote{100-ppv} is never accepted).
The \dQuote{youden} and \dQuote{closest.topleft} are weighted with \code{r},
according to the value of the \code{best.weights} argument. See the
\dQuote{Best thresholds} section above for more details.
For \code{ret}, the single value \dQuote{all} can be used to return
every coordinate available.
}
} % details
\value{
A \code{data.frame} with \code{ret} as columns and as many rows as
given by \code{x}.
In all cases where \code{input="specificity"} or \code{input="sensitivity"}
and interpolation was required, threshold is returned as \code{NA}.
Note that if giving a character as \code{x} (\dQuote{all},
\dQuote{local maximas} or \dQuote{best}), you cannot predict the dimension of
the return value. Even \dQuote{best} may return more than one value (for
example if the ROC curve is below the identity line, both extreme points).
}
\references{
Neil J. Perkins, Enrique F. Schisterman (2006) ``The Inconsistency of "Optimal" Cutpoints
Obtained using Two Criteria based on the Receiver Operating
Characteristic Curve''. \emph{American Journal of Epidemiology}
\bold{163}(7), 670--675. DOI: \doi{10.1093/aje/kwj063}.
Xavier Robin, Natacha Turck, Alexandre Hainard, \emph{et al.}
(2011) ``pROC: an open-source package for R and S+ to analyze and
compare ROC curves''. \emph{BMC Bioinformatics}, \bold{7}, 77.
DOI: \doi{10.1186/1471-2105-12-77}.
W. J. Youden (1950) ``Index for rating diagnostic tests''. \emph{Cancer},
\bold{3}, 32--35. DOI:
\doi{10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3}.
}
\seealso{
\code{\link{roc}}, \code{\link{ci.coords}}
}
\examples{
# Create a ROC curve:
data(aSAH)
roc.s100b <- roc(aSAH$outcome, aSAH$s100b, percent = TRUE)
# Get the coordinates of S100B threshold 0.55
coords(roc.s100b, 0.55)
# Get the coordinates at 50\% sensitivity
coords(roc=roc.s100b, x=50, input="sensitivity")
# Can be abbreviated:
coords(roc.s100b, 50, "se")
# Works with smoothed ROC curves
coords(smooth(roc.s100b), 90, "specificity")
# Get the sensitivities for all thresholds
cc <- coords(roc.s100b, "all", ret="sensitivity")
print(cc$sensitivity)
# Get the best threshold
coords(roc.s100b, "best", ret="threshold")
# Get the best threshold according to different methods
roc.ndka <- roc(aSAH$outcome, aSAH$ndka, percent=TRUE)
coords(roc.ndka, "best", ret="threshold",
best.method="youden") # default
coords(roc.ndka, "best", ret="threshold",
best.method="closest.topleft")
# and with different weights
coords(roc.ndka, "best", ret="threshold",
best.method="youden", best.weights=c(50, 0.2))
coords(roc.ndka, "best", ret="threshold",
best.method="closest.topleft", best.weights=c(5, 0.2))
# This is available with the plot.roc function too:
plot(roc.ndka, print.thres="best", print.thres.best.method="youden",
print.thres.best.weights=c(50, 0.2))
# Return more values:
coords(roc.s100b, "best", ret=c("threshold", "specificity", "sensitivity", "accuracy",
"precision", "recall"))
# Return all values
coords(roc.s100b, "best", ret = "all")
# You can use coords to plot for instance a sensitivity + specificity vs. cut-off diagram
plot(specificity + sensitivity ~ threshold,
coords(roc.ndka, "all"),
type = "l", log="x",
subset = is.finite(threshold))
# Plot the Precision-Recall curve
plot(precision ~ recall,
coords(roc.ndka, "all", ret = c("recall", "precision")),
type="l", ylim = c(0, 100))
# Alternatively plot the curve with TPR and FPR instead of SE/SP
# (identical curve, only the axis change)
plot(tpr ~ fpr,
coords(roc.ndka, "all", ret = c("tpr", "fpr")),
type="l")
}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{roc}
pROC/man/cov.Rd 0000644 0001762 0000144 00000031770 15040443562 012721 0 ustar ligges users \encoding{UTF-8}
\name{cov.roc}
\alias{cov}
\alias{cov.default}
\alias{cov.auc}
\alias{cov.smooth.roc}
\alias{cov.roc}
\title{
Covariance of two paired ROC curves
}
\description{
This function computes the covariance between the AUC of two correlated (or paired) ROC
curves.
}
\usage{
cov(...)
\S3method{cov}{default}(...)
\S3method{cov}{auc}(roc1, roc2, ...)
\S3method{cov}{smooth.roc}(roc1, roc2, ...)
\S3method{cov}{roc}(roc1, roc2, method=c("delong", "bootstrap", "obuchowski"),
reuse.auc=TRUE, boot.n=2000, boot.stratified=TRUE, boot.return=FALSE,
progress=NULL, parallel=FALSE, ...)
}
\arguments{
\item{roc1, roc2}{the two ROC curves on which to compute the covariance. Either
\dQuote{\link{roc}}, \dQuote{\link{auc}} or
\dQuote{\link{smooth.roc}} objects (types can be mixed as long as
the original ROC curve are paired).
}
\item{method}{the method to use, either \dQuote{delong} or
\dQuote{bootstrap}. The first letter is
sufficient. If omitted, the appropriate method is selected as
explained in details.
}
\item{reuse.auc}{if \code{TRUE} (default) and the \dQuote{roc} objects
contain an \dQuote{auc} field, re-use these specifications for the
test. See details.
}
\item{boot.n}{for \code{method="bootstrap"} only: the number of
bootstrap replicates or permutations. Default: \var{2000}.
}
\item{boot.stratified}{for \code{method="bootstrap"} only:
should the bootstrap be stratified (same number
of cases/controls in each replicate than in the original sample) or
not. Default: \var{TRUE}.
}
\item{boot.return}{
if \var{TRUE} and \code{method="bootstrap"}, also return the
bootstrapped values. See the \dQuote{Value} section for
more details.
}
\item{progress}{DEPRECATED. A value other than \code{NULL} will produce
a warning. The argument will be removed in a future version.}
\item{parallel}{DEPRECATED. A value other than \code{FALSE} will produce a warning.
The argument will be removed in a future version.}
\item{\dots}{further arguments passed to or from other methods,
especially arguments for \code{cov.roc} when calling \code{cov},
\code{cov.auc} or \code{cov.smooth.roc}. Arguments for
\code{\link{auc}} (if \code{reuse.auc=FALSE}) and
\code{\link{txtProgressBar}} (only \code{char} and \code{style}) if
applicable.
}
}
\details{
This function computes the covariance between the AUC of two
correlated (or paired, according to the detection of \code{\link{are.paired}}) ROC
curves. It is typically called with the two \link{roc} objects of
interest. Two methods are available: \dQuote{delong} and
\dQuote{bootstrap} (see \dQuote{Computational
details} section below).
The default is to use \dQuote{delong} method except with
partial AUC and smoothed curves where \dQuote{bootstrap} is employed.
Using \dQuote{delong} for partial AUC and smoothed ROCs is not
supported.
For \link[=smooth.roc]{smoothed ROC curves}, smoothing is performed again at each
bootstrap replicate with the parameters originally provided.
If a density smoothing was performed with user-provided
\code{density.cases} or \code{density.controls} the bootstrap cannot
be performed and an error is issued.
\code{cov.default} forces the usage of the
\code{\link[stats:cor]{cov}} function in the \pkg{stats} package, so
that other code relying on \code{cov} should continue to function
normally.
}
\section{AUC specification}{
To compute the covariance of the AUC of the ROC curves, \code{cov} needs a specification of the
AUC. The specification is defined by:
\enumerate{
\item the \dQuote{auc} field in the \dQuote{\link{roc}} objects if
\code{reuse.auc} is set to \code{TRUE} (default)
\item passing the specification to \code{\link{auc}} with \dots
(arguments \code{partial.auc}, \code{partial.auc.correct} and
\code{partial.auc.focus}). In this case, you must ensure either that
the \code{\link{roc}} object do not contain an \code{auc} field (if
you called \code{\link{roc}} with \code{auc=FALSE}), or set
\code{reuse.auc=FALSE}.
}
If \code{reuse.auc=FALSE} the \code{\link{auc}} function will always
be called with \code{\dots} to determine the specification, even if
the \dQuote{\link{roc}} objects do contain an \code{auc} field.
As well if the \dQuote{\link{roc}} objects do not contain an \code{auc}
field, the \code{\link{auc}} function will always be called with
\code{\dots} to determine the specification.
Warning: if the roc object passed to roc.test contains an \code{auc}
field and \code{reuse.auc=TRUE}, \link{auc} is not called and
arguments such as \code{partial.auc} are silently ignored.
}
\section{Computation details}{
With \code{method="bootstrap"}, the processing is done as follow:
\enumerate{
\item \code{boot.n} bootstrap replicates are drawn from the
data. If \code{boot.stratified} is \var{TRUE}, each replicate contains
exactly the same number of controls and cases than the original
sample, otherwise if \var{FALSE} the numbers can vary.
\item for each bootstrap replicate, the AUC of the two ROC curves
are computed and stored.
\item the variance (as per \code{\link{var.roc}}) of the resampled
AUCs and their covariance are assessed in a single bootstrap pass.
\item The following formula is used to compute the final covariance:
\eqn{Var[AUC1] + Var[AUC2] - 2cov[AUC1,AUC2]}
}
With \code{method="delong"}, the processing is done as described in
Hanley and Hajian-Tilaki (1997) using the algorithm by Sun and Xu (2014).
With \code{method="obuchowski"}, the processing is done as described
in Obuchowski and McClish (1997), Table 1 and Equation 5, p. 1531. The
computation of \eqn{g} for partial area under the ROC curve is
modified as:
\deqn{expr1 * (2 * pi * expr2) ^ {(-1)} * (-expr4) - A * B * expr1 * (2 * pi * expr2^3) ^ {(-1/2)} * expr3}.
}
\section{Binormality assumption}{
The \dQuote{obuchowski} method makes the assumption that the data is binormal.
If the data shows a deviation from this assumption, it might help to
normalize the data first (that is, before calling \code{\link{roc}}),
for example with quantile normalization:
\preformatted{
norm.x <- qnorm(rank(x)/(length(x)+1))
cov(roc(response, norm.x, ...), ...)
}
\dQuote{delong} and \dQuote{bootstrap} methods make no such assumption.
}
\value{
The numeric value of the covariance.
If \code{boot.return=TRUE} and \code{method="bootstrap"}, an attribute
\code{resampled.values} is set with the resampled (bootstrapped)
values. It contains a matrix with the columns representing the two ROC
curves, and the rows the \code{boot.n} bootstrap replicates.
}
\section{Errors}{
If \code{density.cases} and \code{density.controls} were provided
for smoothing, the error \dQuote{Cannot compute the covariance on ROC
curves smoothed with density.controls and density.cases.} is
issued.
}
\section{Warnings}{
If \dQuote{auc} specifications are different in both roc objects, the
warning \dQuote{Different AUC specifications in the ROC
curves. Enforcing the inconsistency, but unexpected results may be
produced.} is issued. Unexpected results may be produced.
If one or both ROC curves are \dQuote{smooth.roc} objects with
different smoothing specifications, the warning
\dQuote{Different smoothing parameters in the ROC curves. Enforcing
the inconsistency, but unexpected results may be produced.} is issued.
This warning can be benign, especially if ROC curves were generated
with \code{roc(\ldots, smooth=TRUE)} with different arguments to other
functions (such as plot), or if you really want to compare two ROC
curves smoothed differently.
If \code{method="delong"} and the AUC specification specifies a
partial AUC, the warning \dQuote{Using DeLong for partial AUC is
not supported. Using bootstrap test instead.} is issued. The
\code{method} argument is ignored and \dQuote{bootstrap} is used instead.
If \code{method="delong"} and the ROC
curve is smoothed, the warning \dQuote{Using DeLong for
smoothed ROCs is not supported. Using bootstrap instead.} is
issued. The \code{method} argument is ignored and \dQuote{bootstrap}
is used instead.
DeLong ignores the direction of the ROC curve so that if two
ROC curves have a different \code{direction}, the warning
\dQuote{"DeLong should not be applied to ROC curves with a different
direction."} is printed. However, the spurious computation is enforced.
If \code{boot.stratified=FALSE} and the sample has a large imbalance between
cases and controls, it could happen that one or more of the replicates
contains no case or control observation, or that there are not enough
points for smoothing, producing a \code{NA} area.
The warning \dQuote{NA value(s) produced during bootstrap were ignored.}
will be issued and the observation will be ignored. If you have a large
imbalance in your sample, it could be safer to keep
\code{boot.stratified=TRUE}.
When both ROC curves have an \code{\link{auc}} of 1 (or 100\%), their covariance will always be null.
This is true for both \dQuote{delong} and \dQuote{bootstrap} and methods. This result is misleading,
as the covariance is of course not null.
A \code{\link{warning}} will be displayed to inform of this condition, and of the misleading output.
}
\section{Messages}{
The covariance can only be computed on paired data. This
assumption is enforced by \code{\link{are.paired}}. If the ROC curves
are not paired, the covariance is \code{0} and the message \dQuote{ROC
curves are unpaired.} is printed. If your ROC curves are paired, make
sure they fit \code{\link{are.paired}} criteria.
}
\references{
Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson
(1988) ``Comparing the areas under two or more correlated receiver
operating characteristic curves: a nonparametric
approach''. \emph{Biometrics} \bold{44}, 837--845.
James A. Hanley and Karim O. Hajian-Tilaki (1997) ``Sampling
variability of nonparametric estimates of the areas under receiver
operating characteristic curves: An update''. \emph{Academic
Radiology} \bold{4}, 49--58. DOI:
\doi{10.1016/S1076-6332(97)80161-4}.
Nancy A. Obuchowski, Donna K. McClish (1997). ``Sample size
determination for diagnostic accurary studies involving binormal ROC
curve indices''. \emph{Statistics in Medicine}, \bold{16}(13),
1529--1542. DOI: \doi{10.1002/(SICI)1097-0258(19970715)16:13<1529::AID-SIM565>3.0.CO;2-H}.
Xu Sun and Weichao Xu (2014) ``Fast Implementation of DeLongs Algorithm for Comparing
the Areas Under Correlated Receiver Operating Characteristic Curves''. \emph{IEEE Signal
Processing Letters}, \bold{21}, 1389--1393.
DOI: \doi{10.1109/LSP.2014.2337313}.
Hadley Wickham (2011) ``The Split-Apply-Combine Strategy for Data Analysis''. \emph{Journal of Statistical Software}, \bold{40}, 1--29.
URL: \doi{10.18637/jss.v040.i01}.
}
\seealso{
\code{\link{roc}}, \code{\link{var.roc}}
}
\examples{
data(aSAH)
# Basic example with 2 roc objects
roc1 <- roc(aSAH$outcome, aSAH$s100b)
roc2 <- roc(aSAH$outcome, aSAH$wfns)
cov(roc1, roc2)
\dontrun{
# The latter used Delong. To use bootstrap:
cov(roc1, roc2, method="bootstrap")
# Decrease boot.n for a faster execution:
cov(roc1, roc2, method="bootstrap", boot.n=1000)
}
# To use Obuchowski:
cov(roc1, roc2, method="obuchowski")
\dontrun{
# Comparison can be done on smoothed ROCs
# Smoothing is re-done at each iteration, and execution is slow
cov(smooth(roc1), smooth(roc2))
}
# or from an AUC (no smoothing)
cov(auc(roc1), roc2)
\dontrun{
# With bootstrap and return.values, one can compute the variances of the
# ROC curves in one single bootstrap run:
cov.rocs <- cov(roc1, roc2, method="bootstrap", boot.return=TRUE)
# var(roc1):
var(attr(cov.rocs, "resampled.values")[,1])
# var(roc2):
var(attr(cov.rocs, "resampled.values")[,2])
}
\dontrun{
# Covariance of partial AUC:
roc3 <- roc(aSAH$outcome, aSAH$s100b, partial.auc=c(1, 0.8), partial.auc.focus="se")
roc4 <- roc(aSAH$outcome, aSAH$wfns, partial.auc=c(1, 0.8), partial.auc.focus="se")
cov(roc3, roc4)
# This is strictly equivalent to:
cov(roc3, roc4, method="bootstrap")
# Alternatively, we could re-use roc1 and roc2 to get the same result:
cov(roc1, roc2, reuse.auc=FALSE, partial.auc=c(1, 0.8), partial.auc.focus="se")
}
# Spurious use of DeLong's test with different direction:
roc5 <- roc(aSAH$outcome, aSAH$s100b, direction="<")
roc6 <- roc(aSAH$outcome, aSAH$s100b, direction=">")
cov(roc5, roc6, method="delong")
## Test data from Hanley and Hajian-Tilaki, 1997
disease.present <- c("Yes", "No", "Yes", "No", "No", "Yes", "Yes", "No",
"No", "Yes", "No", "No", "Yes", "No", "No")
field.strength.1 <- c(1, 2, 5, 1, 1, 1, 2, 1, 2, 2, 1, 1, 5, 1, 1)
field.strength.2 <- c(1, 1, 5, 1, 1, 1, 4, 1, 2, 2, 1, 1, 5, 1, 1)
roc7 <- roc(disease.present, field.strength.1)
roc8 <- roc(disease.present, field.strength.2)
# Assess the covariance:
cov(roc7, roc8)
\dontrun{
# With bootstrap:
cov(roc7, roc8, method="bootstrap")
}
}
\keyword{multivariate}
\keyword{nonparametric}
\keyword{utilities}
\keyword{roc}
pROC/man/has.partial.auc.Rd 0000644 0001762 0000144 00000002141 13607143106 015073 0 ustar ligges users \encoding{UTF-8}
\name{has.partial.auc}
\alias{has.partial.auc}
\alias{has.partial.auc.smooth.roc}
\alias{has.partial.auc.auc}
\alias{has.partial.auc.roc}
\title{
Does the ROC curve have a partial AUC?
}
\description{
This function determines if the ROC curve has a partial AUC.
}
\usage{
has.partial.auc(roc)
\S3method{has.partial.auc}{auc}(roc)
\S3method{has.partial.auc}{smooth.roc}(roc)
\S3method{has.partial.auc}{roc}(roc)
}
\arguments{
\item{roc}{the ROC curve to check.}
}
\value{
\code{TRUE} if the AUC is a partial AUC, \code{FALSE} otherwise.
If the AUC is not defined (i. e. if roc was called with \code{AUC=FALSE}), returns \code{NULL}.
}
\seealso{
\code{\link{auc}}
}
\examples{
data(aSAH)
# Full AUC
roc1 <- roc(aSAH$outcome, aSAH$s100b)
has.partial.auc(roc1)
has.partial.auc(auc(roc1))
has.partial.auc(smooth(roc1))
# Partial AUC
roc2 <- roc(aSAH$outcome, aSAH$s100b, partial.auc = c(1, 0.9))
has.partial.auc(roc2)
has.partial.auc(smooth(roc2))
# No AUC
roc3 <- roc(aSAH$outcome, aSAH$s100b, auc = FALSE)
has.partial.auc(roc3)
}
\keyword{programming}
\keyword{logic}
\keyword{roc}
pROC/man/aSAH.Rd 0000644 0001762 0000144 00000003523 14114130125 012666 0 ustar ligges users \encoding{UTF-8}
\name{aSAH}
\docType{data}
\alias{aSAH}
\title{
Subarachnoid hemorrhage data
}
\description{
This dataset summarizes several clinical and one laboratory variable
of 113 patients with an aneurysmal subarachnoid hemorrhage.
}
\usage{aSAH}
\format{A data.frame containing 113 observations of 7 variables.}
\source{
Natacha Turck, Laszlo Vutskits, Paola Sanchez-Pena, Xavier
Robin, Alexandre Hainard, Marianne Gex-Fabry, Catherine Fouda, Hadiji
Bassem, Markus Mueller, Frédérique Lisacek, Louis Puybasset and
Jean-Charles Sanchez (2010) ``A multiparameter panel method for outcome
prediction following aneurysmal subarachnoid hemorrhage''.
\emph{Intensive Care Medicine} \bold{36}(1), 107--115. DOI:
\doi{10.1007/s00134-009-1641-y}.
}
\seealso{
Other examples can be found in all the documentation pages of this
package:
\code{\link{roc}}, \code{\link{auc}}, \code{\link{ci}},
\code{\link{ci.auc}}, \code{\link{ci.se}}, \code{\link{ci.sp}},
\code{\link{ci.thresholds}}, \code{\link{coords}},
\code{\link{plot.ci}}, \code{\link{plot.roc}},
\code{\link{print.roc}}, \code{\link{roc.test}} and
\code{\link{smooth}}.
An example analysis with pROC is shown in:
Xavier Robin, Natacha Turck, Alexandre Hainard, \emph{et al.}
(2011) ``pROC: an open-source package for R and S+ to analyze and
compare ROC curves''. \emph{BMC Bioinformatics}, \bold{7}, 77.
DOI: \doi{10.1186/1471-2105-12-77}
}
\examples{
# load the dataset
data(aSAH)
# Gender, outcome and set
with(aSAH, table(gender, outcome))
# Age
with(aSAH, by(age, outcome, mean))
with(aSAH, by(age, outcome,
function(x) sprintf("mean: \%.1f (+/- \%.1f), median: \%.1f (\%i-\%i)",
mean(x), sd(x), median(x), min(x), max(x))))
# WFNS score
with(aSAH, table(wfns=ifelse(wfns<=2, "1-2", "3-4-5"), outcome))
}
\keyword{datasets}
pROC/man/geom_polygon_auc.roc.Rd 0000644 0001762 0000144 00000003017 15040443562 016233 0 ustar ligges users \encoding{UTF-8}
\name{geom_polygon_auc}
\alias{geom_polygon_auc.auc}
\alias{geom_polygon_auc.roc}
\alias{geom_polygon_auc.smooth.roc}
\alias{geom_polygon_auc}
\title{
Add an AUC polygon to a ggroc plot
}
\description{
EXPERIMENTAL - Add an AUC polygon to a ggroc plot.
}
\usage{
\S3method{geom_polygon_auc}{auc}(data, legacy.axes = FALSE, ...)
\S3method{geom_polygon_auc}{roc}(data, ...)
\S3method{geom_polygon_auc}{smooth.roc}(data, ...)
}
\arguments{
\item{data}{a roc object from the \link{roc} function, same as the one
used to build the ggroc initially.
}
\item{legacy.axes}{must match the value given to \code{ggroc}.
}
\item{...}{additional aesthetics for \code{\link[ggplot2:geom_polygon]{geom_polygon}}
to set: \code{alpha}, \code{colour}, \code{linetype} and \code{linewidth}.
}
}
\details{
This is highly experimental and may change in the future.
}
\seealso{
\code{\link{ggroc}}
}
\examples{
# Create a ROC curve:
data(aSAH)
roc.s100b <- roc(aSAH$outcome, aSAH$s100b)
roc.s100b.percent <- roc(aSAH$outcome, aSAH$s100b, percent = TRUE)
ggroc(roc.s100b) + geom_polygon_auc(roc.s100b$auc)
# legacy.axes must be repeated
ggroc(roc.s100b.percent, legacy.axes=TRUE) + geom_polygon_auc(roc.s100b.percent, legacy.axes=TRUE)
# Partial AUCs
auc.s100b.partial.sp <- auc(roc.s100b, partial.auc = c(0.9, 1))
auc.s100b.partial.se <- auc(roc.s100b, partial.auc = c(0.8, 0.9), partial.auc.focus="se")
ggroc(roc.s100b) + geom_polygon_auc(auc.s100b.partial.sp)
ggroc(roc.s100b) + geom_polygon_auc(auc.s100b.partial.se)
}
pROC/man/multiclass.Rd 0000644 0001762 0000144 00000015162 14114130125 014274 0 ustar ligges users \encoding{UTF-8}
\name{multiclass.roc}
\alias{multiclass.roc}
\alias{multiclass.roc.default}
\alias{multiclass.roc.formula}
\title{
Multi-class AUC
}
\description{
This function builds builds multiple ROC curve to compute the
multi-class AUC as defined by Hand and Till.
}
\usage{
multiclass.roc(...)
\S3method{multiclass.roc}{formula}(formula, data, ...)
\S3method{multiclass.roc}{default}(response, predictor,
levels=base::levels(as.factor(response)),
percent=FALSE, direction = c("auto", "<", ">"), ...)
}
\arguments{
\item{response}{a factor, numeric or character vector of
responses (true class), typically encoded with 0 (controls) and 1 (cases), as in
\code{\link{roc}}.
}
\item{predictor}{either a numeric vector, containing the value of each
observation, as in \code{\link{roc}}, or, a matrix giving the decision value
(e.g. probability) for each class.
}
\item{formula}{a formula of the type \code{response~predictor}.}
\item{data}{a matrix or data.frame containing the variables in the
formula. See \code{\link{model.frame}} for more details.}
\item{levels}{the value of the response for controls and cases
respectively. In contrast with \code{levels} argument to
\code{\link{roc}}, all the levels are used and
\link[=combn]{combined} to compute the multiclass AUC.
}
\item{percent}{if the sensitivities, specificities and AUC must be
given in percent (\code{TRUE}) or in fraction (\code{FALSE}, default).
}
\item{direction}{in which direction to make the comparison?
\dQuote{auto} (default for univariate curves):
automatically define in which group the
median is higher and take the direction accordingly.
Not available for multivariate curves.
\dQuote{>} (default for multivariate curves):
if the predictor values for the control group are
higher than the values of the case group (controls > t >= cases).
\dQuote{<}: if the predictor values for the control group are lower
or equal than the values of the case group (controls < t <= cases).
}
\item{...}{further arguments passed to \code{\link{roc}}.
}
}
\details{
This function performs multiclass AUC as defined by Hand and Till
(2001). A multiclass AUC is a mean of several \code{\link{auc}} and
cannot be plotted. Only AUCs can be computed for such curves.
Confidence intervals, standard deviation, smoothing and
comparison tests are not implemented.
The \code{multiclass.roc} function can handle two types of datasets: uni- and multi-variate.
In the univariate case, a single \code{predictor} vector is passed
and all the combinations of responses are assessed.
I the multivariate case, a \code{\link{matrix}} or \code{\link{data.frame}}
is passed as \code{predictor}. The columns must be named according to the
levels of the \code{response}.
This function has been much less tested than the rest of the package and
is more subject to bugs. Please report them if you find one.
}
\value{
If \code{predictor} is a vector, a list of class \dQuote{multiclass.roc}
(univariate) or \dQuote{mv.multiclass.roc} (multivariate),
with the following fields:
\item{auc}{if called with \code{auc=TRUE}, a numeric of class \dQuote{auc} as
defined in \code{\link{auc}}. Note that this is not the standard AUC
but the multi-class AUC as defined by Hand and Till.
}
\item{ci}{if called with \code{ci=TRUE}, a numeric of class \dQuote{ci} as
defined in \code{\link{ci}}.
}
\item{response}{the response vector as passed in argument. If
\code{NA} values were removed, a \code{na.action} attribute similar
to \code{\link{na.omit}} stores the row numbers.
}
\item{predictor}{the predictor vector as passed in argument. If
\code{NA} values were removed, a \code{na.action} attribute similar
to \code{\link{na.omit}} stores the row numbers.
}
\item{levels}{the levels of the response as defined in argument.}
\item{percent}{if the sensitivities, specificities and AUC are
reported in percent, as defined in argument.
}
\item{call}{how the function was called. See \code{\link{match.call}} for
more details.
}
}
\section{Warnings}{
If \code{response} is an ordered factor and one of the levels
specified in \code{levels} is missing, a warning is issued and the
level is ignored.
}
\references{
David J. Hand and Robert J. Till (2001). A Simple Generalisation of
the Area Under the ROC Curve for Multiple Class Classification
Problems. \emph{Machine Learning} \bold{45}(2), p. 171--186. DOI:
\doi{10.1023/A:1010920819831}.
}
\seealso{
\code{\link{auc}}
}
\examples{
####
# Examples for a univariate decision value
####
data(aSAH)
# Basic example
multiclass.roc(aSAH$gos6, aSAH$s100b)
# Produces an innocuous warning because one level has no observation
# Select only 3 of the aSAH$gos6 levels:
multiclass.roc(aSAH$gos6, aSAH$s100b, levels=c(3, 4, 5))
# Give the result in percent
multiclass.roc(aSAH$gos6, aSAH$s100b, percent=TRUE)
####
# Examples for multivariate decision values (e.g. class probabilities)
####
\dontrun{
# Example with a multinomial log-linear model from nnet
# We use the iris dataset and split into a training and test set
requireNamespace("nnet")
data(iris)
iris.sample <- sample(1:150)
iris.train <- iris[iris.sample[1:75],]
iris.test <- iris[iris.sample[76:150],]
mn.net <- nnet::multinom(Species ~ ., iris.train)
# Use predict with type="prob" to get class probabilities
iris.predictions <- predict(mn.net, newdata=iris.test, type="prob")
head(iris.predictions)
# This can be used directly in multiclass.roc:
multiclass.roc(iris.test$Species, iris.predictions)
}
# Let's see an other example with an artificial dataset
n <- c(100, 80, 150)
responses <- factor(c(rep("X1", n[1]), rep("X2", n[2]), rep("X3", n[3])))
# construct prediction matrix: one column per class
preds <- lapply(n, function(x) runif(x, 0.4, 0.6))
predictor <- as.matrix(data.frame(
"X1" = c(preds[[1]], runif(n[2] + n[3], 0, 0.7)),
"X2" = c(runif(n[1], 0.1, 0.4), preds[[2]], runif(n[3], 0.2, 0.8)),
"X3" = c(runif(n[1] + n[2], 0.3, 0.7), preds[[3]])
))
multiclass.roc(responses, predictor)
# One can change direction , partial.auc, percent, etc:
multiclass.roc(responses, predictor, direction = ">")
multiclass.roc(responses, predictor, percent = TRUE,
partial.auc = c(100, 90), partial.auc.focus = "se")
# Limit set of levels
multiclass.roc(responses, predictor, levels = c("X1", "X2"))
# Use with formula. Here we need a data.frame to store the responses as characters
data <- cbind(as.data.frame(predictor), "response" = responses)
multiclass.roc(response ~ X1+X3, data)
}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{roc}
pROC/man/plot.roc.Rd 0000644 0001762 0000144 00000033561 14114130125 013657 0 ustar ligges users \encoding{UTF-8}
\name{plot.roc}
\alias{plot}
\alias{plot.roc}
\alias{plot.roc.roc}
\alias{plot.smooth.roc}
\alias{plot.roc.smooth.roc}
\alias{plot.roc.default}
\alias{plot.roc.formula}
\title{
Plot a ROC curve
}
\description{
This function plots a ROC curve. It can accept many arguments to
tweak the appearance of the plot. Two syntaxes are possible: one
object of class \dQuote{\link{roc}}, or either two vectors (response, predictor) or a
formula (response~predictor) as in the \code{\link{roc}} function.
}
\usage{
\S3method{plot}{roc}(x, ...)
\S3method{plot}{smooth.roc}(x, ...)
\S3method{plot.roc}{roc}(x, add=FALSE, reuse.auc=TRUE,
axes=TRUE, legacy.axes=FALSE,
# Generic arguments for par:
xlim=if(x$percent){c(100, 0)} else{c(1, 0)},
ylim=if(x$percent){c(0, 100)} else{c(0, 1)},
xlab=ifelse(x$percent, ifelse(legacy.axes, "100 - Specificity (\%)", "Specificity (\%)"),
ifelse(legacy.axes, "1 - Specificity", "Specificity")),
ylab=ifelse(x$percent, "Sensitivity (\%)", "Sensitivity"),
asp=1,
mar=c(4, 4, 2, 2)+.1,
mgp=c(2.5, 1, 0),
# col, lty and lwd for the ROC line only
col=par("col"),
lty=par("lty"),
lwd=2,
type="l",
# Identity line
identity=!add,
identity.col="darkgrey",
identity.lty=1,
identity.lwd=1,
# Print the thresholds on the plot
print.thres=FALSE,
print.thres.pch=20,
print.thres.adj=c(-.05,1.25),
print.thres.col="black",
print.thres.pattern=ifelse(x$percent, "\%.1f (\%.1f\%\%, \%.1f\%\%)", "\%.3f (\%.3f, \%.3f)"),
print.thres.cex=par("cex"),
print.thres.pattern.cex=print.thres.cex,
print.thres.best.method=NULL,
print.thres.best.weights=c(1, 0.5),
# Print the AUC on the plot
print.auc=FALSE,
print.auc.pattern=NULL,
print.auc.x=ifelse(x$percent, 50, .5),
print.auc.y=ifelse(x$percent, 50, .5),
print.auc.adj=c(0,1),
print.auc.col=col,
print.auc.cex=par("cex"),
# Grid
grid=FALSE,
grid.v={if(is.logical(grid) && grid[1]==TRUE)
{seq(0, 1, 0.1) * ifelse(x$percent, 100, 1)}
else if(is.numeric(grid))
{seq(0, ifelse(x$percent, 100, 1), grid[1])} else {NULL}},
grid.h={if (length(grid) == 1) {grid.v}
else if (is.logical(grid) && grid[2]==TRUE)
{seq(0, 1, 0.1) * ifelse(x$percent, 100, 1)}
else if(is.numeric(grid))
{seq(0, ifelse(x$percent, 100, 1), grid[2])} else {NULL}},
grid.lty=3,
grid.lwd=1,
grid.col="#DDDDDD",
# Polygon for the AUC
auc.polygon=FALSE,
auc.polygon.col="gainsboro",
auc.polygon.lty=par("lty"),
auc.polygon.density=NULL,
auc.polygon.angle=45,
auc.polygon.border=NULL,
# Polygon for the maximal AUC possible
max.auc.polygon=FALSE,
max.auc.polygon.col="#EEEEEE",
max.auc.polygon.lty=par("lty"),
max.auc.polygon.density=NULL,
max.auc.polygon.angle=45,
max.auc.polygon.border=NULL,
# Confidence interval
ci=!is.null(x$ci),
ci.type=c("bars", "shape", "no"),
ci.col=ifelse(ci.type=="bars", par("fg"), "gainsboro"),
...)
\S3method{plot.roc}{formula}(x, data, subset, na.action, ...)
\S3method{plot.roc}{default}(x, predictor, ...)
\S3method{plot.roc}{smooth.roc}(x, ...)
}
\arguments{
\item{x}{a roc object from the \link{roc} function (for plot.roc.roc),
a formula (for plot.roc.formula) or a response vector (for
plot.roc.default).
}
\item{predictor, data}{arguments for the \link{roc} function.}
\item{subset,na.action}{arguments for \code{\link{model.frame}}}
\item{add}{if TRUE, the ROC curve will be added to an existing
plot. If FALSE (default), a new plot will be created.
}
\item{reuse.auc}{if \code{TRUE} (default) and the \dQuote{roc} object
contains an \dQuote{auc} field, re-use these specifications for the
plot (specifically \code{print.auc}, \code{auc.polygon} and
\code{max.auc.polygon} arguments). See details.
}
\item{axes}{a logical indicating if the plot axes must be drawn.}
\item{legacy.axes}{a logical indicating if the specificity axis (x
axis) must be plotted as as decreasing \dQuote{specificity}
(\code{FALSE}, the default) or increasing \dQuote{1 - specificity}
(\code{TRUE}) as in most legacy software. This affects only the
axis, not the plot coordinates.}
\item{xlim, ylim, xlab, ylab, asp, mar, mgp}{Generic arguments for the
plot. See \link{plot} and \link{plot.window} for more details. Only
used if \code{add=FALSE}.
}
\item{col,lty, lwd}{color, line type and line width for the ROC
curve. See \link{par} for more details.
}
\item{type}{type of plotting as in \code{\link{plot}}.}
\item{identity}{logical: whether or not the identity line (no discrimination
line) must be displayed. Default: only on new plots.
}
\item{identity.col, identity.lty, identity.lwd}{color, line type and
line width for the identity line. Used only if identity=TRUE. See
\link{par} for more details.
}
\item{print.thres}{Should a selected set of thresholds be displayed on
the ROC curve? \code{FALSE}, \code{NULL} or \dQuote{no}: no threshold is
displayed. \code{TRUE} or \dQuote{best}: the threshold with the
highest sum sensitivity + specificity is plotted (this might be more
than one threshold). \dQuote{all}: all the points of the ROC
curve. \dQuote{local maximas}: all the local maximas. Numeric
vector: direct definition of the thresholds to display.
Note that on a smoothed ROC curve, only \dQuote{best} is supported.
}
\item{print.thres.pch, print.thres.adj, print.thres.col,
print.thres.cex}{the plotting character (pch), text string
adjustment (adj), color (col) and character expansion factor (cex)
parameters for the printing of the thresholds. See \link{points} and
\link{par} for more details.
}
\item{print.thres.pattern}{the text pattern for the thresholds, as a
\link{sprintf} format. Three numerics are passed to sprintf:
threshold, specificity, sensitivity.
}
\item{print.thres.pattern.cex}{the character expansion factor (cex) for the
threshold text pattern. See \link{par} for more details.
}
\item{print.thres.best.method, print.thres.best.weights}{if
\code{print.thres="best"} or \code{print.thres=TRUE}, what method must be used to determine which
threshold is the best. See argument \code{best.method} and \code{best.weights} to
\code{\link{coords}} for more details.
}
\item{print.auc}{boolean. Should the numeric value of AUC be printed
on the plot?
}
\item{print.auc.pattern}{the text pattern for the AUC, as a
\link{sprintf} format. If NULL, a reasonable value is computed that
takes partial AUC, CI and percent into account. If the CI
of the AUC was computed, three numerics are passed to
sprintf: AUC, lower CI bound, higher CI bound. Otherwise, only AUC is
passed.
}
\item{print.auc.x, print.auc.y}{x and y position for the printing of
the AUC.
}
\item{print.auc.adj, print.auc.cex, print.auc.col}{the text
adjustment, character expansion factor and color for the printing of
the AUC. See \link{par} for more details.
}
\item{grid}{boolean or numeric vector of length 1 or 2. Should a
background grid be added to the plot? Numeric: show a grid with the
specified interval between each line; Logical: show the grid or
not. Length 1: same values are taken for horizontal and vertical
lines. Length 2: grid value for vertical (grid[1]) and horizontal
(grid[2]). Note that these values are used to compute grid.v and
grid.h. Therefore if you specify a grid.h and grid.v, it will be
ignored.
}
\item{grid.v, grid.h}{numeric. The x and y values at which a vertical
or horizontal line (respectively) must be drawn. NULL if no line
must be added.
}
\item{grid.lty, grid.lwd, grid.col}{the line type (lty), line width (lwd) and
color (col) of the lines of the grid. See \link{par} for
more details. Note that you can pass vectors of length 2, in which
case it specifies the vertical (1) and horizontal (2) lines.
}
\item{auc.polygon}{boolean. Whether or not to display the area as a
polygon.
}
\item{auc.polygon.col, auc.polygon.lty, auc.polygon.density,
auc.polygon.angle, auc.polygon.border}{color (col), line type
(lty), density, angle and border for the AUC polygon. See
\code{\link{polygon}} and \code{\link{par}} for more details.
}
\item{max.auc.polygon}{boolean. Whether or not to display the maximal
possible area as a polygon.
}
\item{max.auc.polygon.col, max.auc.polygon.lty, max.auc.polygon.density,
max.auc.polygon.angle, max.auc.polygon.border}{color (col), line type
(lty), density, angle and border for the maximum AUC polygon. See
\code{\link{polygon}} and \code{\link{par}} for more details.
}
\item{ci}{boolean. Should we plot the confidence intervals?}
\item{ci.type, ci.col}{\code{type} and \code{col} arguments for
\code{\link{plot.ci}}. The special value \dQuote{no} disables the plotting of confidence intervals.
}
\item{\dots}{further arguments passed to or from other methods,
especially arguments for \code{\link{roc}} and \code{plot.roc.roc} when calling
\code{plot.roc.default} or \code{plot.roc.formula}. Note that the
\code{plot} argument for \code{\link{roc}} is not allowed.
Arguments for \code{\link{auc}} and graphical functions
\code{\link{plot}}, \code{\link{abline}}, \code{\link{polygon}},
\code{\link{points}}, \code{\link{text}} and \code{\link{plot.ci}}
if applicable.
}
}
\details{
This function is typically called from \code{\link{roc}} when plot=TRUE (not by
default). \code{plot.roc.formula} and \code{plot.roc.default} are convenience methods
that build the ROC curve (with the \code{\link{roc}} function) before
calling \code{plot.roc.roc}. You can pass them arguments for both
\code{\link{roc}} and \code{plot.roc.roc}. Simply use \code{plot.roc}
that will dispatch to the correct method.
The plotting is done in the following order:
\enumerate{
\item A new plot is created if \code{add=FALSE}.
\item The grid is added if \code{grid.v} and \code{grid.h} are not NULL.
\item The maximal AUC polygon is added if \code{max.auc.polygon=TRUE}.
\item The CI shape is added if \code{ci=TRUE}, \code{ci.type="shape"} and \code{x$ci} isn't a \dQuote{ci.auc}.
\item The AUC polygon is added if \code{auc.polygon=TRUE}.
\item The identity line if \code{identity=TRUE}.
\item The actual ROC line is added.
\item The CI bars are added if \code{ci=TRUE}, \code{ci.type="bars"} and \code{x$ci} isn't a \dQuote{ci.auc}.
\item The selected thresholds are printed if \code{print.thres} is \code{TRUE} or numeric.
\item The AUC is printed if \code{print.auc=TRUE}.
}
Graphical functions are called with \link{suppressWarnings}.
}
\section{AUC specification}{
For \code{print.auc}, \code{auc.polygon} and \code{max.auc.polygon}
arguments, an AUC specification is
required. By default, the total AUC is plotted, but you may want a
partial AUCs. The specification is defined by:
\enumerate{
\item the \dQuote{auc} field in the \dQuote{\link{roc}} object if
\code{reuse.auc} is set to \code{TRUE} (default). It is naturally
inherited from any call to \code{\link{roc}} and fits most cases.
\item passing the specification to \code{\link{auc}} with \dots
(arguments \code{partial.auc}, \code{partial.auc.correct} and
\code{partial.auc.focus}). In this case, you must ensure either that
the \code{\link{roc}} object do not contain an \code{auc} field (if
you called \code{\link{roc}} with \code{auc=FALSE}), or set
\code{reuse.auc=FALSE}.
}
If \code{reuse.auc=FALSE} the \code{\link{auc}} function will always
be called with \code{\dots} to determine the specification, even if
the \dQuote{\link{roc}} object do contain an \code{auc} field.
As well if the \dQuote{\link{roc}} object do not contain an \code{auc}
field, the \code{\link{auc}} function will always be called with
\code{\dots} to determine the specification.
Warning: if the roc object passed to plot.roc contains an \code{auc}
field and \code{reuse.auc=TRUE}, \link{auc} is not called and
arguments such as \code{partial.auc} are silently ignored.
}
\value{
This function returns a list of class \dQuote{roc} invisibly. See \link{roc} for more details.
}
\references{
Xavier Robin, Natacha Turck, Alexandre Hainard, \emph{et al.}
(2011) ``pROC: an open-source package for R and S+ to analyze and
compare ROC curves''. \emph{BMC Bioinformatics}, \bold{7}, 77.
DOI: \doi{10.1186/1471-2105-12-77}.
}
\seealso{
\code{\link{roc}}, \code{\link{auc}}, \code{\link{ci}}
}
\examples{
# Create a few ROC curves:
data(aSAH)
roc.s100b <- roc(aSAH$outcome, aSAH$s100b)
roc.wfns <- roc(aSAH$outcome, aSAH$wfns)
roc.ndka <- roc(aSAH$outcome, aSAH$wfns)
# Simple example:
plot(roc.s100b)
# Add a smoothed ROC:
plot(smooth(roc.s100b), add=TRUE, col="blue")
legend("bottomright", legend=c("Empirical", "Smoothed"),
col=c(par("fg"), "blue"), lwd=2)
# With more options:
plot(roc.s100b, print.auc=TRUE, auc.polygon=TRUE, grid=c(0.1, 0.2),
grid.col=c("green", "red"), max.auc.polygon=TRUE,
auc.polygon.col="lightblue", print.thres=TRUE)
# To plot a different partial AUC, we need to ignore the existing value
# with reuse.auc=FALSE:
plot(roc.s100b, print.auc=TRUE, auc.polygon=TRUE, partial.auc=c(1, 0.8),
partial.auc.focus="se", grid=c(0.1, 0.2), grid.col=c("green", "red"),
max.auc.polygon=TRUE, auc.polygon.col="lightblue",
print.thres=TRUE, print.thres.adj = c(1, -1),
reuse.auc=FALSE)
# Add a second ROC curve to the previous plot:
plot(roc.wfns, add=TRUE)
# Plot some thresholds, add them to the same plot
plot(roc.ndka, print.thres="best", print.thres.best.method="youden")
plot(roc.ndka, print.thres="best", print.thres.best.method="closest.topleft",
add = TRUE)
plot(roc.ndka, print.thres="best", print.thres.best.method="youden",
print.thres.best.weights=c(50, 0.2),
print.thres.adj = c(1.1, 1.25),
add = TRUE)
}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{aplot}
\keyword{hplot}
\keyword{roc}
pROC/man/ci.auc.Rd 0000644 0001762 0000144 00000023424 15040443562 013271 0 ustar ligges users \encoding{UTF-8}
\name{ci.auc}
\alias{ci.auc}
\alias{ci.auc.auc}
\alias{ci.auc.default}
\alias{ci.auc.formula}
\alias{ci.auc.roc}
\alias{ci.auc.smooth.roc}
\alias{ci.auc.multiclass.roc}
\alias{ci.auc.multiclass.auc}
\title{
Compute the confidence interval of the AUC
}
\description{
This function computes the confidence interval (CI) of an area under
the curve (AUC).
}
\usage{
# ci.auc(...)
\S3method{ci.auc}{roc}(roc, conf.level=0.95, method=c("delong",
"bootstrap"), boot.n = 2000, boot.stratified = TRUE, reuse.auc=TRUE,
progress = NULL, parallel=FALSE, ...)
\S3method{ci.auc}{smooth.roc}(smooth.roc, conf.level=0.95, boot.n=2000,
boot.stratified=TRUE, reuse.auc=TRUE,
progress = NULL, parallel=FALSE, ...)
\S3method{ci.auc}{auc}(auc, ...)
\S3method{ci.auc}{multiclass.roc}(multiclass.roc, ...)
\S3method{ci.auc}{multiclass.auc}(multiclass.auc, ...)
\S3method{ci.auc}{auc}(auc, ...)
\S3method{ci.auc}{formula}(formula, data, ...)
\S3method{ci.auc}{default}(response, predictor, ...)
}
\arguments{
\item{roc, smooth.roc}{a \dQuote{roc} object from the
\code{\link{roc}} function, or a \dQuote{smooth.roc} object from the
\code{\link[=smooth.roc]{smooth}} function.
}
\item{auc}{an \dQuote{auc} object from the \code{\link{auc}} function.}
\item{multiclass.roc, multiclass.auc}{not implemented.}
\item{response, predictor}{arguments for the \code{\link{roc}} function.}
\item{formula, data}{a formula (and possibly a data object) of type
response~predictor for the \code{\link{roc}} function.
}
\item{conf.level}{the width of the confidence interval as [0,1], never
in percent. Default: 0.95, resulting in a 95\% CI.
}
\item{method}{the method to use, either \dQuote{delong} or
\dQuote{bootstrap}. The first letter is sufficient. If omitted, the
appropriate method is selected as explained in details.
}
\item{boot.n}{the number of bootstrap replicates. Default: 2000.}
\item{boot.stratified}{should the bootstrap be stratified (default, same number
of cases/controls in each replicate than in the original sample) or
not.
}
\item{reuse.auc}{if \code{TRUE} (default) and the \dQuote{roc} object
contains an \dQuote{auc} field, re-use these specifications for the
test. If false, use optional \code{\dots} arguments to
\code{\link{auc}}. See details.
}
\item{progress}{DEPRECATED. A value other than \code{NULL} will produce
a warning. The argument will be removed in a future version.}
\item{parallel}{DEPRECATED. A value other than \code{FALSE} will produce a warning.
The argument will be removed in a future version.}
\item{\dots}{further arguments passed to or from other methods,
especially arguments for \code{\link{roc}} and \code{roc.test.roc}
when calling \code{roc.test.default} or \code{roc.test.formula}.
Arguments for \code{\link{auc}}
and \code{\link{txtProgressBar}} (only \code{char} and \code{style})
if applicable.
}
}
\details{
This function computes the CI of an AUC. Two methods are available:
\dQuote{delong} and \dQuote{bootstrap} with the parameters defined in \dQuote{roc$auc} to
compute a CI. When it is called with two vectors (response, predictor)
or a formula (response~predictor) arguments, the \code{\link{roc}}
function is called to build the ROC curve first.
The default is to use the \dQuote{delong} method, except for comparison of
partial AUC and smoothed curves, where \code{bootstrap} is used. Using
\dQuote{delong} for partial AUC and smoothed ROCs is not supported.
With \code{method="bootstrap"}, the function calls \code{\link{auc}}
\code{boot.n} times. For more details about the bootstrap, see the Bootstrap section in
\link[=pROC-package]{this package's documentation}.
For \link[=smooth.roc]{smoothed ROC curves}, smoothing is performed again at each
bootstrap replicate with the parameters originally provided.
If a density smoothing was performed with user-provided
\code{density.cases} or \code{density.controls} the bootstrap cannot
be performed and an error is issued.
With \code{method="delong"}, the variance of the AUC is computed as
defined by DeLong \emph{et al.} (1988) using the algorithm by Sun and Xu (2014)
and the CI is deduced with \code{\link{qnorm}}.
CI of multiclass ROC curves and AUC is not implemented yet. Attempting to call these
methods returns an error.
}
\section{AUC specification}{
The comparison of the CI needs a specification of the AUC. This allows
to compute the CI for full or partial AUCs. The specification is defined by:
\enumerate{
\item the \dQuote{auc} field in the \dQuote{\link{roc}} object if
\code{reuse.auc} is set to \code{TRUE} (default). It is naturally
inherited from any call to \code{\link{roc}} and fits most cases.
\item passing the specification to \code{\link{auc}} with \dots
(arguments \code{partial.auc}, \code{partial.auc.correct} and
\code{partial.auc.focus}). In this case, you must ensure either that
the \code{\link{roc}} object do not contain an \code{auc} field (if
you called \code{\link{roc}} with \code{auc=FALSE}), or set
\code{reuse.auc=FALSE}.
}
If \code{reuse.auc=FALSE} the \code{\link{auc}} function will always
be called with \code{\dots} to determine the specification, even if
the \dQuote{\link{roc}} object do contain an \code{auc} field.
As well if the \dQuote{\link{roc}} object do not contain an \code{auc}
field, the \code{\link{auc}} function will always be called with
\code{\dots} to determine the specification.
Warning: if the roc object passed to ci contains an \code{auc}
field and \code{reuse.auc=TRUE}, \link{auc} is not called and
arguments such as \code{partial.auc} are silently ignored.
}
\section{Warnings}{
If \code{method="delong"} and the AUC specification specifies a
partial AUC, the warning \dQuote{Using DeLong's test for partial AUC is
not supported. Using bootstrap test instead.} is issued. The
\code{method} argument is ignored and \dQuote{bootstrap} is used
instead.
If \code{boot.stratified=FALSE} and the sample has a large imbalance between
cases and controls, it could happen that one or more of the replicates
contains no case or control observation, or that there are not enough
points for smoothing, producing a \code{NA} area.
The warning \dQuote{NA value(s) produced during bootstrap were ignored.}
will be issued and the observation will be ignored. If you have a large
imbalance in your sample, it could be safer to keep
\code{boot.stratified=TRUE}.
}
\section{Errors}{
If \code{density.cases} and \code{density.controls} were provided
for smoothing, the error \dQuote{Cannot compute the statistic on ROC
curves smoothed with density.controls and density.cases.} is issued.
}
\value{
A numeric vector of length 3 and class \dQuote{ci.auc}, \dQuote{ci} and \dQuote{numeric} (in this order), with the lower
bound, the median and the upper bound of the CI, and the following attributes:
\item{conf.level}{the width of the CI, in fraction.}
\item{method}{the method employed.}
\item{boot.n}{the number of bootstrap replicates.}
\item{boot.stratified}{whether or not the bootstrapping was stratified.}
\item{auc}{an object of class \dQuote{\link{auc}} stored for reference about the
compued AUC details (partial, percent, ...)}
The \code{aucs} item is not included in this list since version 1.2 for
consistency reasons.
}
\references{
James Carpenter and John Bithell (2000) ``Bootstrap condence intervals:
when, which, what? A practical guide for medical statisticians''.
\emph{Statistics in Medicine} \bold{19}, 1141--1164.
DOI: \doi{10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F}.
Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson
(1988) ``Comparing the areas under two or more correlated receiver
operating characteristic curves: a nonparametric
approach''. \emph{Biometrics} \bold{44}, 837--845.
Xu Sun and Weichao Xu (2014) ``Fast Implementation of DeLongs Algorithm for Comparing
the Areas Under Correlated Receiver Operating Characteristic Curves''. \emph{IEEE Signal
Processing Letters}, \bold{21}, 1389--1393.
DOI: \doi{10.1109/LSP.2014.2337313}.
Xavier Robin, Natacha Turck, Alexandre Hainard, \emph{et al.}
(2011) ``pROC: an open-source package for R and S+ to analyze and
compare ROC curves''. \emph{BMC Bioinformatics}, \bold{7}, 77.
DOI: \doi{10.1186/1471-2105-12-77}.
Hadley Wickham (2011) ``The Split-Apply-Combine Strategy for Data Analysis''. \emph{Journal of Statistical Software}, \bold{40}, 1--29.
URL: \doi{10.18637/jss.v040.i01}.
}
\seealso{
\code{\link{roc}}, \code{\link{auc}}, \code{\link{ci}}
}
\examples{
# Create a ROC curve:
data(aSAH)
roc1 <- roc(aSAH$outcome, aSAH$s100b)
## Basic example ##
ci.auc(roc1)
# You can also write:
ci(roc1)
ci(auc(roc1))
## More options ##
# Partial AUC and customized bootstrap:
\dontrun{
ci.auc(roc1,
conf.level=0.9,
partial.auc=c(1, .8), partial.auc.focus="se", partial.auc.correct=TRUE,
boot.n=10000, stratified=FALSE)}\dontshow{
ci.auc(roc1,
conf.level=0.9,
partial.auc=c(1, .8), partial.auc.focus="se", partial.auc.correct=TRUE,
boot.n=10, stratified=FALSE)}
# Note that the following will NOT give a CI of the partial AUC:
\dontrun{
ci.auc(roc1,
partial.auc=c(1, .8), partial.auc.focus="se", partial.auc.correct=FALSE)}
# This is because rocobj$auc is not a partial AUC and reuse.auc = TRUE by default.
# You can overcome this problem by passing an AUC instead:
auc1 <- auc(roc1, partial.auc=c(1, .8), partial.auc.focus="se",
partial.auc.correct=FALSE)
\dontrun{
ci.auc(auc1)}\dontshow{ci.auc(auc1, boot.n = 10)}
## On smoothed ROC curves with bootstrap ##
\dontrun{
ci.auc(smooth(roc1, method="density"))}\dontshow{
ci.auc(smooth(roc1, method="density"), boot.n = 10)}
}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{roc}
pROC/man/ci.sp.Rd 0000644 0001762 0000144 00000014420 15040443562 013137 0 ustar ligges users \encoding{UTF-8}
\name{ci.sp}
\alias{ci.sp}
\alias{ci.sp.default}
\alias{ci.sp.formula}
\alias{ci.sp.roc}
\alias{ci.sp.smooth.roc}
\title{
Compute the confidence interval of specificities at given sensitivities
}
\description{
This function computes the confidence interval (CI) of the specificity
at the given sensitivity points.
}
\usage{
# ci.sp(...)
\S3method{ci.sp}{roc}(roc, sensitivities = seq(0, 1, .1) * ifelse(roc$percent,
100, 1), conf.level=0.95, boot.n=2000, boot.stratified=TRUE,
progress = NULL, parallel=FALSE, ...)
\S3method{ci.sp}{smooth.roc}(smooth.roc, sensitivities = seq(0, 1, .1) *
ifelse(smooth.roc$percent, 100, 1), conf.level=0.95, boot.n=2000,
boot.stratified=TRUE, progress = NULL, parallel=FALSE, ...)
\S3method{ci.sp}{formula}(formula, data, ...)
\S3method{ci.sp}{default}(response, predictor, ...)
}
\arguments{
\item{roc, smooth.roc}{a \dQuote{roc} object from the
\code{\link{roc}} function, or a \dQuote{smooth.roc} object from the
\code{\link[=smooth.roc]{smooth}} function.
}
\item{response, predictor}{arguments for the \code{\link{roc}} function.}
\item{formula, data}{a formula (and possibly a data object) of type
response~predictor for the \code{\link{roc}} function.
}
\item{sensitivities}{on which sensitivities to evaluate the CI.
}
\item{conf.level}{the width of the confidence interval as [0,1], never
in percent. Default: 0.95, resulting in a 95\% CI.
}
\item{boot.n}{the number of bootstrap replicates. Default: 2000.}
\item{boot.stratified}{should the bootstrap be stratified (default, same number
of cases/controls in each replicate than in the original sample) or
not.
}
\item{progress}{DEPRECATED. A value other than \code{NULL} will produce
a warning.The argument will be removed in a future version.}
\item{parallel}{DEPRECATED. A value other than \code{FALSE} will produce a warning.
The argument will be removed in a future version.}
\item{\dots}{further arguments passed to or from other methods,
especially arguments for \code{\link{roc}} and \code{ci.sp.roc}
when calling \code{ci.sp.default} or \code{ci.sp.formula}.
Arguments for \code{\link{txtProgressBar}} (only
\code{char} and \code{style}) if applicable.
}
}
\details{
\code{ci.sp.formula} and \code{ci.sp.default} are convenience methods
that build the ROC curve (with the \code{\link{roc}} function) before
calling \code{ci.sp.roc}. You can pass them arguments for both
\code{\link{roc}} and \code{ci.sp.roc}. Simply use \code{ci.sp}
that will dispatch to the correct method.
The \code{ci.sp.roc} function creates \code{boot.n} bootstrap replicate of the ROC
curve, and evaluates the specificity at sensitivities
given by the \code{sensitivities} argument. Then it computes the
confidence interval as the percentiles given by \code{conf.level}.
For more details about the bootstrap, see the Bootstrap section in
\link[=pROC-package]{this package's documentation}.
For \link[=smooth.roc]{smoothed ROC curves}, smoothing is performed again at each
bootstrap replicate with the parameters originally provided.
If a density smoothing was performed with user-provided
\code{density.cases} or \code{density.controls} the bootstrap cannot
be performed and an error is issued.
}
\section{Warnings}{
If \code{boot.stratified=FALSE} and the sample has a large imbalance between
cases and controls, it could happen that one or more of the replicates
contains no case or control observation, or that there are not enough
points for smoothing, producing a \code{NA} area.
The warning \dQuote{NA value(s) produced during bootstrap were ignored.}
will be issued and the observation will be ignored. If you have a large
imbalance in your sample, it could be safer to keep
\code{boot.stratified=TRUE}.
}
\section{Errors}{
If \code{density.cases} and \code{density.controls} were provided
for smoothing, the error \dQuote{Cannot compute the statistic on ROC
curves smoothed with density.controls and density.cases.} is issued.
}
\value{
A matrix of class \dQuote{ci.sp}, \dQuote{ci} and \dQuote{matrix} (in this order)
containing the given specificities. Row (names) are the
sensitivities, the first column the lower bound, the 2nd column the
median and the 3rd column the upper bound.
Additionally, the list has the following attributes:
\item{conf.level}{the width of the CI, in fraction.}
\item{boot.n}{the number of bootstrap replicates.}
\item{boot.stratified}{whether or not the bootstrapping was stratified.}
\item{sensitivities}{the sensitivities as given in argument.}
\item{roc}{the object of class \dQuote{\link{roc}} that was used to
compute the CI.
}
}
\references{
James Carpenter and John Bithell (2000) ``Bootstrap condence intervals:
when, which, what? A practical guide for medical statisticians''.
\emph{Statistics in Medicine} \bold{19}, 1141--1164.
DOI: \doi{10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F}.
Tom Fawcett (2006) ``An introduction to ROC analysis''. \emph{Pattern
Recognition Letters} \bold{27}, 861--874. DOI:
\doi{10.1016/j.patrec.2005.10.010}.
Xavier Robin, Natacha Turck, Alexandre Hainard, \emph{et al.}
(2011) ``pROC: an open-source package for R and S+ to analyze and
compare ROC curves''. \emph{BMC Bioinformatics}, \bold{7}, 77.
DOI: \doi{10.1186/1471-2105-12-77}.
Hadley Wickham (2011) ``The Split-Apply-Combine Strategy for Data Analysis''. \emph{Journal of Statistical Software}, \bold{40}, 1--29.
URL: \doi{10.18637/jss.v040.i01}.
}
\seealso{
\code{\link{roc}},
\code{\link{ci}},
\code{\link{ci.se}},
\code{\link{plot.ci}}
}
\examples{
# Create a ROC curve:
data(aSAH)
roc1 <- roc(aSAH$outcome, aSAH$s100b)
## Basic example ##
\dontrun{
ci.sp(roc1)}\dontshow{ci.sp(roc1, boot.n = 10)}
## More options ##
# Customized bootstrap and sensitivities:
\dontrun{
ci.sp(roc1, c(.95, .9, .85), boot.n=10000, conf.level=0.9, stratified=FALSE)}\dontshow{
ci.sp(roc1, c(.95, .9, .85), boot.n=10, conf.level=0.9, stratified=FALSE)}
## Plotting the CI ##
ci1 <- ci.sp(roc1, boot.n = 10)
plot(roc1)
plot(ci1)
## On smoothed ROC curves with bootstrap ##
\dontrun{
ci.sp(smooth(roc1, method="density"))}\dontshow{
ci.sp(smooth(roc1, method="density"), boot.n = 10)}
}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{roc}
pROC/man/ci.Rd 0000644 0001762 0000144 00000010174 15040443562 012520 0 ustar ligges users \encoding{UTF-8}
\name{ci}
\alias{ci}
\alias{ci.default}
\alias{ci.formula}
\alias{ci.roc}
\alias{ci.smooth.roc}
\alias{ci.multiclass.roc}
\alias{ci.multiclass.auc}
\title{
Compute the confidence interval of a ROC curve
}
\description{
This function computes the confidence interval (CI) of a ROC curve. The
\code{of} argument controls the type of CI that will be computed.
}
\usage{
ci(...)
\S3method{ci}{roc}(roc, of = c("auc", "thresholds", "sp", "se", "coords"), ...)
\S3method{ci}{smooth.roc}(smooth.roc, of = c("auc", "sp", "se", "coords"), ...)
\S3method{ci}{multiclass.roc}(multiclass.roc, of = "auc", ...)
\S3method{ci}{multiclass.auc}(multiclass.auc, of = "auc", ...)
\S3method{ci}{formula}(formula, data, ...)
\S3method{ci}{default}(response, predictor, ...)
}
\arguments{
\item{roc, smooth.roc}{a \dQuote{roc} object from the
\code{\link{roc}} function, or a \dQuote{smooth.roc} object from the
\code{\link[=smooth.roc]{smooth}} function.
}
\item{multiclass.roc, multiclass.auc}{not implemented.}
\item{response, predictor}{arguments for the \code{\link{roc}} function.}
\item{formula, data}{a formula (and possibly a data object) of type
response~predictor for the \code{\link{roc}} function.
}
\item{of}{The type of confidence interval. One of \dQuote{auc},
\dQuote{thresholds}, \dQuote{sp}, \dQuote{se} or \dQuote{coords}. Note that
confidence interval on \dQuote{thresholds} are not available for
smoothed ROC curves.
}
\item{\dots}{further arguments passed to or from other methods,
especially \code{\link{auc}}, \code{\link{roc}}, and the specific
\code{ci} functions \code{\link{ci.auc}}, \code{\link{ci.se}},
\code{\link{ci.sp}} and \code{\link{ci.thresholds}}.
}
}
\details{
\code{ci.formula} and \code{ci.default} are convenience methods
that build the ROC curve (with the \code{\link{roc}} function) before
calling \code{ci.roc}. You can pass them arguments for both
\code{\link{roc}} and \code{ci.roc}. Simply use \code{ci}
that will dispatch to the correct method.
This function is typically called from \code{\link{roc}} when \code{ci=TRUE} (not by
default). Depending on the \code{of} argument, the specific
\code{ci} functions \code{\link{ci.auc}}, \code{\link{ci.thresholds}},
\code{\link{ci.sp}}, \code{\link{ci.se}} or \code{\link{ci.coords}} are called.
When the ROC curve has an \code{\link{auc}} of 1 (or 100\%), the confidence interval will always be null
(there is no interval). This is true for both \dQuote{delong} and \dQuote{bootstrap} methods that can
not properly assess the variance in this case. This result is misleading, as the variance is of course not null.
A \code{\link{warning}} will be displayed to inform of this condition, and of the misleading output.
CI of multiclass ROC curves and AUC is not implemented yet. Attempting to call these
methods returns an error.
}
\value{
The return value of the specific \code{ci} functions
\code{\link{ci.auc}}, \code{\link{ci.thresholds}}, \code{\link{ci.sp}}, \code{\link{ci.se}} or \code{\link{ci.coords}}, depending on the
\code{of} argument.
}
\references{
Xavier Robin, Natacha Turck, Alexandre Hainard, \emph{et al.}
(2011) ``pROC: an open-source package for R and S+ to analyze and
compare ROC curves''. \emph{BMC Bioinformatics}, \bold{7}, 77.
DOI: \doi{10.1186/1471-2105-12-77}.
}
\seealso{
\code{\link{roc}}, \code{\link{auc}}, \code{\link{ci.auc}},
\code{\link{ci.thresholds}}, \code{\link{ci.sp}}, \code{\link{ci.se}}, \code{\link{ci.coords}}
}
\examples{
# Create a ROC curve:
data(aSAH)
roc1 <- roc(aSAH$outcome, aSAH$s100b)
## AUC ##
ci(roc1)
# this is equivalent to:
ci(roc1, of = "auc")
# or:
ci.auc(roc1)
## Coordinates ##
\dontrun{
# Thresholds
ci(roc1, of = "thresholds")
ci(roc1, of = "thresholds", thresholds = "all")
ci(roc1, of = "thresholds", thresholds = 0.51)
# equivalent to:
ci.thresholds(roc1, thresholds = 0.51)
# SE/SP
ci(roc1, of = "sp", sensitivities = c(.95, .9, .85))
ci.sp(roc1)
ci(roc1, of = "se")
ci.se(roc1)
# Arbitrary coordinates
ci(roc1, of = "coords", "best")
ci.coords(roc1, 0.51, "threshold")}
}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{roc}
pROC/man/ci.thresholds.Rd 0000644 0001762 0000144 00000014704 15040443562 014701 0 ustar ligges users \encoding{UTF-8}
\name{ci.thresholds}
\alias{ci.thresholds}
\alias{ci.thresholds.default}
\alias{ci.thresholds.formula}
\alias{ci.thresholds.roc}
\alias{ci.thresholds.smooth.roc}
\title{
Compute the confidence interval of thresholds
}
\description{
This function computes the confidence interval (CI) of the sensitivity
and specificity of the thresholds given in argument.
}
\usage{
# ci.thresholds(...)
\S3method{ci.thresholds}{roc}(roc, conf.level=0.95, boot.n=2000,
boot.stratified=TRUE, thresholds = "local maximas",
progress = NULL, parallel=FALSE, ...)
\S3method{ci.thresholds}{formula}(formula, data, ...)
\S3method{ci.thresholds}{smooth.roc}(smooth.roc, ...)
\S3method{ci.thresholds}{default}(response, predictor, ...)
}
\arguments{
\item{roc}{a \dQuote{roc} object from the \code{\link{roc}} function.}
\item{smooth.roc}{not available for \link[=smooth.roc]{smoothed} ROC
curves, available only to catch the error and provide a clear error
message.
}
\item{response, predictor}{arguments for the \code{\link{roc}} function.}
\item{formula, data}{a formula (and possibly a data object) of type
response~predictor for the \code{\link{roc}} function.
}
\item{conf.level}{the width of the confidence interval as [0,1], never
in percent. Default: 0.95, resulting in a 95\% CI.
}
\item{boot.n}{the number of bootstrap replicates. Default: 2000.}
\item{boot.stratified}{should the bootstrap be stratified (default, same number
of cases/controls in each replicate than in the original sample) or
not.
}
\item{thresholds}{on which thresholds to evaluate the CI. Either the
numeric values of the thresholds, a logical vector (as index of
\code{roc$thresholds}) or a character \dQuote{all}, \dQuote{local
maximas} or \dQuote{best} that will be used to determine the threshold(s)
on the supplied curve with \code{\link{coords}} (not on the resampled curves).
}
\item{progress}{DEPRECATED. A value other than \code{NULL} will produce
a warning. The argument will be removed in a future version.}
\item{parallel}{DEPRECATED. A value other than \code{FALSE} will produce a warning.
The argument will be removed in a future version.}
\item{\dots}{further arguments passed to or from other methods,
especially arguments for \code{\link{roc}} and \code{ci.thresholds.roc}
when calling \code{ci.thresholds.default} or \code{ci.thresholds.formula}.
Arguments for \code{\link{txtProgressBar}} (only
\code{char} and \code{style}) if applicable.
Arguments \code{best.method} and \code{best.weights} to \code{\link{coords}}.
}
}
\details{
\code{ci.thresholds.formula} and \code{ci.thresholds.default} are convenience methods
that build the ROC curve (with the \code{\link{roc}} function) before
calling \code{ci.thresholds.roc}. You can pass them arguments for both
\code{\link{roc}} and \code{ci.thresholds.roc}. Simply use \code{ci.thresholds}
that will dispatch to the correct method.
This function creates \code{boot.n} bootstrap replicate of the ROC
curve, and evaluates the sensitivity and specificity at thresholds
given by the \code{thresholds} argument. Then it computes the
confidence interval as the percentiles given by \code{conf.level}.
A threshold given as a \code{logical} vector or \code{character} is converted to the corresponding numeric vector once
\emph{using the supplied ROC curve}, and not at each bootstrap iteration. See \code{\link{ci.coords}} for the latter behaviour.
For more details about the bootstrap, see the Bootstrap section in
\link[=pROC-package]{this package's documentation}.
}
\section{Warnings}{
If \code{boot.stratified=FALSE} and the sample has a large imbalance between
cases and controls, it could happen that one or more of the replicates
contains no case or control observation, producing a \code{NA} area.
The warning \dQuote{NA value(s) produced during bootstrap were ignored.}
will be issued and the observation will be ignored. If you have a large
imbalance in your sample, it could be safer to keep
\code{boot.stratified=TRUE}.
}
\value{
A list of length 2 and class \dQuote{ci.thresholds}, \dQuote{ci} and \dQuote{list} (in this order), with the confidence
intervals of the CI and the following items:
\item{specificity}{a matrix of CI for the specificity. Row (names) are the
thresholds, the first column the lower bound, the 2nd column the
median and the 3rd column the upper bound.
}
\item{sensitivity}{same than specificity.}
Additionally, the list has the following attributes:
\item{conf.level}{the width of the CI, in fraction.}
\item{boot.n}{the number of bootstrap replicates.}
\item{boot.stratified}{whether or not the bootstrapping was stratified.}
\item{thresholds}{the thresholds, as given in argument.}
\item{roc}{the object of class \dQuote{\link{roc}} that was used to
compute the CI.
}
}
\references{
James Carpenter and John Bithell (2000) ``Bootstrap condence intervals:
when, which, what? A practical guide for medical statisticians''.
\emph{Statistics in Medicine} \bold{19}, 1141--1164.
DOI: \doi{10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F}.
Tom Fawcett (2006) ``An introduction to ROC analysis''. \emph{Pattern
Recognition Letters} \bold{27}, 861--874. DOI:
\doi{10.1016/j.patrec.2005.10.010}.
Xavier Robin, Natacha Turck, Alexandre Hainard, \emph{et al.}
(2011) ``pROC: an open-source package for R and S+ to analyze and
compare ROC curves''. \emph{BMC Bioinformatics}, \bold{7}, 77.
DOI: \doi{10.1186/1471-2105-12-77}.
Hadley Wickham (2011) ``The Split-Apply-Combine Strategy for Data Analysis''. \emph{Journal of Statistical Software}, \bold{40}, 1--29.
URL: \doi{10.18637/jss.v040.i01}.
}
\seealso{
\code{\link{roc}},
\code{\link{ci}}
}
\examples{
data(aSAH)
# Create a ROC curve:
data(aSAH)
roc1 <- roc(aSAH$outcome, aSAH$s100b)
## Basic example ##
# Compute CI of all local maxima thresholds
\dontrun{
ci.thresholds(roc1)}\dontshow{ci.thresholds(roc1, boot.n = 10)}
## More options ##
# Customized bootstrap and thresholds:
\dontrun{
ci.thresholds(roc1,
thresholds=c(0.5, 1, 2),
boot.n=10000, conf.level=0.9, stratified=FALSE)}\dontshow{
ci.thresholds(roc1,
thresholds=c(0.5, 1, 2),
boot.n=10, conf.level=0.9, stratified=FALSE)}
## Plotting the CI ##
\dontrun{
ci1 <- ci.thresholds(roc1)}\dontshow{
ci1 <- ci.thresholds(roc1, boot.n = 10)}
plot(roc1)
plot(ci1)
}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{roc}
pROC/man/smooth.Rd 0000644 0001762 0000144 00000041006 15040443562 013434 0 ustar ligges users \encoding{UTF-8}
\name{smooth}
\alias{smooth}
\alias{smooth.roc}
\alias{smooth.smooth.roc}
\alias{smooth.default}
\title{
Smooth a ROC curve
}
\description{
This function smoothes a ROC curve of numeric predictor. By default, a
binormal smoothing is performed, but density or custom smoothings are
supported.
}
\usage{
smooth(...)
\S3method{smooth}{default}(...)
\S3method{smooth}{roc}(roc,
method=c("binormal", "density", "fitdistr", "logcondens",
"logcondens.smooth"), n=512, bw = "nrd0", density=NULL,
density.controls=density, density.cases=density,
start=NULL, start.controls=start, start.cases=start,
reuse.auc=TRUE, reuse.ci=FALSE, ...)
\S3method{smooth}{smooth.roc}(smooth.roc, ...)
}
\arguments{
\item{roc, smooth.roc}{a \dQuote{roc} object from the
\code{\link{roc}} function, or a \dQuote{smooth.roc} object from the
\code{\link[=smooth.roc]{smooth}} function.
}
\item{method}{\dQuote{binormal}, \dQuote{density}, \dQuote{fitdistr},
\dQuote{logcondens}, \dQuote{"logcondens.smooth"}.
}
\item{n}{
the number of equally spaced points where the smoothed curve will be
calculated.
}
\item{bw}{
if \code{method="density"} and \code{density.controls} and
\code{density.cases} are not provided, \code{bw} is passed to
\code{\link{density}} to determine the bandwidth of the
density Can be a character string (\dQuote{nrd0}, \dQuote{nrd},
\dQuote{ucv}, \dQuote{bcv} or \dQuote{SJ}, but any name
\link[base:match.fun]{matching} a function prefixed with \dQuote{bw.} is
supported) or a numeric value, as described in \code{\link{density}}.
Defaults to \dQuote{\link[stats:bandwidth]{nrd0}}.
}
\item{density, density.controls, density.cases}{if
\code{method="density"}, a numeric value of density (over the y
axis) or a function returning a density (such as
\code{\link{density}}. If \code{method="fitdistr"}, a \code{densfun}
argument for \code{\link[MASS]{fitdistr}}.
If the value is different for control and case observations,
\code{density.controls} and \code{density.cases} can be employed
instead, otherwise \code{density} will be propagated to both
\code{density.controls} and \code{density.cases}.
}
\item{start, start.controls, start.cases}{if
\code{method="fitdistr"}, optionnal \code{start}
arguments for . \code{start.controls}
and \code{start.cases} allows to specify different distributions for
controls and cases.
}
\item{reuse.auc, reuse.ci}{if \code{TRUE} (default for reuse.auc) and the \dQuote{roc} objects
contain \dQuote{auc} or \dQuote{ci} fields, re-use these
specifications to regenerate \code{\link{auc}} or \code{\link{ci}}
on the smoothed ROC curve with the original parameters. If
\code{FALSE}, the object returned will not contain
\dQuote{auc} or \dQuote{ci} fields. It is currently not possible to
redefine \code{\link{auc}} and \code{\link{ci}} options directly: you need to call \code{\link{auc}} or
\code{\link{ci}} later for that.
}
\item{\dots}{further arguments passed to or from other methods, and
especially to \code{\link{density}} (only \code{cut}, \code{adjust},
and \code{kernel}, plus \code{window} for compatibility with S+) and
\code{\link[MASS]{fitdistr}}.
}
}
\details{
If \code{method="binormal"}, a linear model is fitted to the quantiles of
the sensitivities and specificities. Smoothed sensitivities and
specificities are then generated from this model on \code{n} points.
This simple approach was found to work well for most ROC curves, but
it may produce hooked smooths in some situations (see in Hanley (1988)).
With \code{method="density"}, the \code{\link{density}}
function is employed to generate a smooth kernel
density of the control and case observations as described by Zhou
\emph{et al.} (1997), unless
\code{density.controls} or \code{density.cases} are provided
directly. \code{bw} can be given to
specify a bandwidth to use with \code{\link{density}}. It can be a
numeric value or a character string (\dQuote{nrd0}, \dQuote{nrd},
\dQuote{ucv}, \dQuote{bcv} or \dQuote{SJ}, but any name
\link[base:match.fun]{matching} a function prefixed with \dQuote{bw.} is
supported). In the case of a character
string, the whole predictor data is employed to determine the numeric
value to use on both controls and cases.
Depending on your data, it might be a good idea to specify the
\code{kernel} argument for \code{\link{density}}. By default,
\dQuote{gaussian} is used, but \dQuote{epanechnikov},
\dQuote{rectangular}, \dQuote{triangular}, \dQuote{biweight},
\dQuote{cosine} and \dQuote{optcosine} are supported. As all the
kernels are symetrical, it might help to normalize the data first
(that is, before calling \code{\link{roc}}), for example with quantile
normalization:
\preformatted{
norm.x <- qnorm(rank(x)/(length(x)+1))
smooth(roc(response, norm.x, ...), ...)
}
Additionally, \code{density} can be a function which must return
either a numeric vector of densities over the y axis or a \link{list}
with a \dQuote{y} item like the \code{\link{density}} function. It
must accept the following input:
\preformatted{
density.fun(x, n, from, to, bw, kernel, ...)
}
It is important to honour \code{n}, \code{from} and \code{to} in order
to have the densities evaluated on the same points for controls and
cases. Failing to do so and returning densities of different length
will produce an error. It is also a good idea to use a constant
smoothing parameter (such as \code{bw}) especially when controls and
cases have a different number of observations, to avoid producing
smoother or rougher densities.
If \code{method="fitdistr"}, the \code{\link[MASS]{fitdistr}}
function from the \pkg{MASS} package is employed to fit parameters for
the density function \code{density} with optionnal start parameters
\code{start}. The density function are fitted
separately in control (\code{density.controls}, \code{start.controls})
and case observations (\code{density.cases},
\code{start.cases}). \code{density} can be one of the character values
allowed by \code{\link[MASS]{fitdistr}} or a density function (such
as \code{\link{dnorm}}, \code{\link{dweibull}}, ...).
The \code{method="logcondens"} and \code{method="logcondens.smooth"} use the
\pkg{logcondens} package to generate a non smoothed or smoothed
(respectively) log-concave density estimate of of the control and case
observation with the \link[logcondens]{logConROC} function.
\code{smooth.default} forces the usage of the
\code{\link[stats]{smooth}} function in the \pkg{stats} package, so
that other code relying on \code{smooth} should continue to function
normally.
Smoothed ROC curves can be passed to smooth again. In this case, the
smoothing is not re-applied on the smoothed ROC curve but the
original \dQuote{\link{roc}} object will be re-used.
Note that a \code{smooth.roc} curve has no threshold.
}
\value{
A list of class \dQuote{smooth.roc} with the following fields:
\item{sensitivities}{the smoothed sensitivities defining the ROC curve.}
\item{specificities}{the smoothed specificities defining the ROC curve.}
\item{percent}{if the sensitivities, specificities and AUC are
reported in percent, as defined in argument.
}
\item{direction}{the direction of the comparison, as defined in argument.}
\item{call}{how the function was called. See \code{\link{match.call}} for
more details.
}
\item{smoothing.args}{a list of the arguments used for the
smoothing. Will serve to apply the smoothing again in further
bootstrap operations.
}
\item{auc}{if the original ROC curve contained an AUC, it is computed
again on the smoothed ROC.
}
\item{ci}{if the original ROC curve contained a CI, it is computed
again on the smoothed ROC.
}
\item{fit.controls, fit.cases}{with \code{method="fitdistr"} only:
the result of \pkg{MASS}'s
\code{\link[MASS]{fitdistr}} function for controls and cases, with an
additional \dQuote{densfun} item indicating the density function, if
possible as character.
}
\item{logcondens}{with \code{method="logcondens"} and \code{method="logcondens.smooth"} only:
the result of \pkg{logcondens}'s \link[logcondens]{logConROC} function.
}
\item{model}{with \code{method="binormal"} only:
the linear model from \code{\link{lm}} used to smooth the ROC curve.
}
\subsection{Attributes}{
Additionally, the original \code{\link{roc}} object is stored as a
\dQuote{roc} attribute.
}
}
\section{Errors}{
The message \dQuote{The 'density' function must return a numeric
vector or a list with a 'y' item.} will be displayed if the
\code{density} function did not return a valid output. The message
\dQuote{Length of 'density.controls' and 'density.cases' differ.}
will be displayed if the returned value differ in length.
Binormal smoothing cannot smooth ROC curve defined by only one
point. Any such attempt will fail with the error \dQuote{ROC curve not
smoothable (not enough points).}.
If the smooth ROC curve was generated by \code{\link{roc}} with
\code{density.controls} and \code{density.cases} numeric arguments, it
cannot be smoothed and the error \dQuote{Cannot smooth a ROC curve
generated directly with numeric 'density.controls' and
'density.cases'.} is produced.
\code{fitdistr} and \code{density} smoothing methods require a
\link{numeric} \code{predictor}. If the ROC curve to smooth was
generated with an ordered factor only binormal smoothing can be
applied and the message \dQuote{ROC curves of ordered predictors can
be smoothed only with binormal smoothing.} is displayed otherwise.
\code{fitdistr}, \code{logcondens} and \code{logcondens.smooth} methods
require additional packages. If not available, the following message
will be displayed with the required command to install the package:
\dQuote{Package ? not available, required with method='?'.
Please install it with 'install.packages("?")'.
}
}
\references{
James E. Hanley (1988) ``The robustness of the ``binormal'' assumptions
used in fitting ROC curves''. \emph{Medical Decision Making} \bold{8}, 197--203.
Lutz Duembgen, Kaspar Rufibach (2011) ``logcondens: Computations Related
to Univariate Log-Concave Density Estimation''. \emph{Journal of Statistical
Software}, \bold{39}, 1--28. URL: \doi{10.18637/jss.v039.i06}.
Xavier Robin, Natacha Turck, Alexandre Hainard, \emph{et al.}
(2011) ``pROC: an open-source package for R and S+ to analyze and
compare ROC curves''. \emph{BMC Bioinformatics}, \bold{7}, 77.
DOI: \doi{10.1186/1471-2105-12-77}.
Kaspar Rufibach (2011) ``A Smooth ROC Curve Estimator Based on Log-Concave Density Estimates''.
\emph{The International Journal of Biostatistics}, \bold{8}, accepted. DOI:
\doi{10.1515/1557-4679.1378}. arXiv:
\href{https://arxiv.org/abs/1103.1787}{arXiv:1103.1787}.
William N. Venables, Brian D. Ripley (2002). ``Modern Applied Statistics with S''. New York, Springer.
\href{http://books.google.ch/books?id=974c4vKurNkC}{Google books}.
Kelly H. Zou, W. J. Hall and David E. Shapiro (1997) ``Smooth
non-parametric receiver operating characteristic (ROC) curves for
continuous diagnostic tests''. \emph{Statistics in Medicine}
\bold{18}, 2143--2156. DOI:
\doi{10.1002/(SICI)1097-0258(19971015)16:19<2143::AID-SIM655>3.0.CO;2-3}.
}
\seealso{
\code{\link{roc}}
CRAN packages \pkg{MASS} and \pkg{logcondens} employed in this function.
}
\examples{
data(aSAH)
## Basic example
rocobj <- roc(aSAH$outcome, aSAH$s100b)
smooth(rocobj)
# or directly with roc()
roc(aSAH$outcome, aSAH$s100b, smooth=TRUE)
# plotting
plot(rocobj)
rs <- smooth(rocobj, method="binormal")
plot(rs, add=TRUE, col="green")
rs2 <- smooth(rocobj, method="density")
plot(rs2, add=TRUE, col="blue")
rs3 <- smooth(rocobj, method="fitdistr", density="lognormal")
plot(rs3, add=TRUE, col="magenta")
if (requireNamespace("logcondens")) {
rs4 <- smooth(rocobj, method="logcondens")
plot(rs4, add=TRUE, col="brown")
rs5 <- smooth(rocobj, method="logcondens.smooth")
plot(rs5, add=TRUE, col="orange")
}
legend("bottomright", legend=c("Empirical", "Binormal", "Density", "Log-normal",
"Log-concave density", "Smoothed log-concave density"),
col=c("black", "green", "blue", "magenta", "brown", "orange"), lwd=2)
## Advanced smoothing
# if we know the distributions are normal with sd=0.1 and an unknown mean:
smooth(rocobj, method="fitdistr", density=dnorm, start=list(mean=1), sd=.1)
# different distibutions for controls and cases:
smooth(rocobj, method="fitdistr", density.controls="normal", density.cases="lognormal")
# with densities
bw <- bw.nrd0(rocobj$predictor)
density.controls <- density(rocobj$controls, from=min(rocobj$predictor) - 3 * bw,
to=max(rocobj$predictor) + 3*bw, bw=bw, kernel="gaussian")
density.cases <- density(rocobj$cases, from=min(rocobj$predictor) - 3 * bw,
to=max(rocobj$predictor) + 3*bw, bw=bw, kernel="gaussian")
smooth(rocobj, method="density", density.controls=density.controls$y,
density.cases=density.cases$y)
# which is roughly what is done by a simple:
smooth(rocobj, method="density")
\dontrun{
## Smoothing artificial ROC curves
rand.unif <- runif(1000, -1, 1)
rand.exp <- rexp(1000)
rand.norm <-
rnorm(1000)
# two normals
roc.norm <- roc(controls=rnorm(1000), cases=rnorm(1000)+1, plot=TRUE)
plot(smooth(roc.norm), col="green", lwd=1, add=TRUE)
plot(smooth(roc.norm, method="density"), col="red", lwd=1, add=TRUE)
plot(smooth(roc.norm, method="fitdistr"), col="blue", lwd=1, add=TRUE)
if (requireNamespace("logcondens")) {
plot(smooth(roc.norm, method="logcondens"), col="brown", lwd=1, add=TRUE)
plot(smooth(roc.norm, method="logcondens.smooth"), col="orange", lwd=1, add=TRUE)
}
legend("bottomright", legend=c("empirical", "binormal", "density", "fitdistr",
"logcondens", "logcondens.smooth"),
col=c(par("fg"), "green", "red", "blue", "brown", "orange"), lwd=c(2, 1, 1, 1))
# deviation from the normality
roc.norm.exp <- roc(controls=rnorm(1000), cases=rexp(1000), plot=TRUE)
plot(smooth(roc.norm.exp), col="green", lwd=1, add=TRUE)
plot(smooth(roc.norm.exp, method="density"), col="red", lwd=1, add=TRUE)
# Wrong fitdistr: normality assumed by default
plot(smooth(roc.norm.exp, method="fitdistr"), col="blue", lwd=1, add=TRUE)
# Correct fitdistr
plot(smooth(roc.norm.exp, method="fitdistr", density.controls="normal",
density.cases="exponential"), col="purple", lwd=1, add=TRUE)
if (requireNamespace("logcondens")) {
plot(smooth(roc.norm.exp, method="logcondens"), col="brown", lwd=1, add=TRUE)
plot(smooth(roc.norm.exp, method="logcondens.smooth"), col="orange", lwd=1, add=TRUE)
}
legend("bottomright", legend=c("empirical", "binormal", "density",
"wrong fitdistr", "correct fitdistr",
"logcondens", "logcondens.smooth"),
col=c(par("fg"), "green", "red", "blue", "purple", "brown", "orange"), lwd=c(2, 1, 1, 1, 1))
# large deviation from the normality
roc.unif.exp <- roc(controls=runif(1000, 2, 3), cases=rexp(1000)+2, plot=TRUE)
plot(smooth(roc.unif.exp), col="green", lwd=1, add=TRUE)
plot(smooth(roc.unif.exp, method="density"), col="red", lwd=1, add=TRUE)
plot(smooth(roc.unif.exp, method="density", bw="ucv"), col="magenta", lwd=1, add=TRUE)
# Wrong fitdistr: normality assumed by default (uniform distributions not handled)
plot(smooth(roc.unif.exp, method="fitdistr"), col="blue", lwd=1, add=TRUE)
if (requireNamespace("logcondens")) {
plot(smooth(roc.unif.exp, method="logcondens"), col="brown", lwd=1, add=TRUE)
plot(smooth(roc.unif.exp, method="logcondens.smooth"), col="orange", lwd=1, add=TRUE)
}
legend("bottomright", legend=c("empirical", "binormal", "density",
"density ucv", "wrong fitdistr",
"logcondens", "logcondens.smooth"),
col=c(par("fg"), "green", "red", "magenta", "blue", "brown", "orange"), lwd=c(2, 1, 1, 1, 1))
}
# 2 uniform distributions with a custom density function
unif.density <- function(x, n, from, to, bw, kernel, ...) {
smooth.x <- seq(from=from, to=to, length.out=n)
smooth.y <- dunif(smooth.x, min=min(x), max=max(x))
return(smooth.y)
}
roc.unif <- roc(controls=runif(1000, -1, 1), cases=runif(1000, 0, 2), plot=TRUE)
s <- smooth(roc.unif, method="density", density=unif.density)
plot(roc.unif)
plot(s, add=TRUE, col="grey")
\dontrun{
# you can bootstrap a ROC curve smoothed with a density function:
ci(s, boot.n=100)
}
}
\keyword{univar}
\keyword{nonparametric}
\keyword{utilities}
\keyword{roc}
\keyword{smooth}
pROC/DESCRIPTION 0000644 0001762 0000144 00000004145 15042637256 012601 0 ustar ligges users Package: pROC
Type: Package
Title: Display and Analyze ROC Curves
Version: 1.19.0.1
Date: 2025-07-31
Encoding: UTF-8
Depends: R (>= 2.14)
Imports: methods, Rcpp (>= 0.11.1)
Suggests: tcltk, MASS, logcondens, testthat, vdiffr, ggplot2, rlang
LinkingTo: Rcpp
Authors@R: c(person("Xavier", "Robin", role = c("cre", "aut"),
email = "pROC-cran@xavier.robin.name",
comment = c(ORCID = "0000-0002-6813-3200")),
person("Natacha", "Turck", role = "aut"),
person("Alexandre", "Hainard", role = "aut"),
person("Natalia", "Tiberti", role = "aut"),
person("Frédérique", "Lisacek", role = "aut"),
person("Jean-Charles", "Sanchez", role = "aut"),
person("Markus", "Müller", role = "aut"),
person("Stefan", "Siegert", role = "ctb",
comment = "Fast DeLong code",
email = "stefan_siegert@gmx.de"),
person("Matthias", "Doering", role = "ctb",
comment = "Hand & Till Multiclass"),
person("Zane", "Billings", role = "ctb",
comment = "DeLong paired test CI"))
Description: Tools for visualizing, smoothing and comparing receiver operating characteristic (ROC curves). (Partial) area under the curve (AUC) can be compared with statistical tests based on U-statistics or bootstrap. Confidence intervals can be computed for (p)AUC or ROC curves.
License: GPL (>= 3)
URL: https://xrobin.github.io/pROC/
BugReports: https://github.com/xrobin/pROC/issues
LazyData: yes
NeedsCompilation: yes
Packaged: 2025-07-31 07:57:35 UTC; xavier
Author: Xavier Robin [cre, aut] (ORCID:
),
Natacha Turck [aut],
Alexandre Hainard [aut],
Natalia Tiberti [aut],
Frédérique Lisacek [aut],
Jean-Charles Sanchez [aut],
Markus Müller [aut],
Stefan Siegert [ctb] (Fast DeLong code),
Matthias Doering [ctb] (Hand & Till Multiclass),
Zane Billings [ctb] (DeLong paired test CI)
Maintainer: Xavier Robin
Repository: CRAN
Date/Publication: 2025-07-31 10:00:14 UTC