ó =ä!Mc @sÆdZddlZddlmZmZddlmZddlmZm Z m Z m Z ddl m Z mZmZmZddd d d d d ddg Zed„ƒZdefd„ƒYZdefd„ƒYZd efd„ƒYZd efd„ƒYZdefd„ƒYZdefd„ƒYZd„Zdefd„ƒYZeƒZ d„Z!defd„ƒYZ"defd „ƒYZ#d efd!„ƒYZ$d e#fd"„ƒYZ%d#„Z&dS($s´ pygments.lexer ~~~~~~~~~~~~~~ Base lexer classes. :copyright: Copyright 2006-2010 by the Pygments team, see AUTHORS. :license: BSD, see LICENSE for details. iÿÿÿÿN(t apply_filterstFilter(tget_filter_by_name(tErrortTexttOthert _TokenType(t get_bool_optt get_int_optt get_list_opttmake_analysatortLexert RegexLexertExtendedRegexLexertDelegatingLexert LexerContexttincludetbygroupstusingtthiscCsdS(Ng((tx((s2/usr/lib/python2.7/site-packages/pygments/lexer.pytst LexerMetacBseZdZd„ZRS(s‚ This metaclass automagically converts ``analyse_text`` methods into static methods which always return float values. cCs9d|kr#t|dƒ|ds(R(t __class__R(R-((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyt__repr__\s   cKs5t|tƒs!t||}n|jj|ƒdS(s8 Add a new stream filter to this lexer. N(t isinstanceRRR'tappend(R-R.R(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR,cscCsdS(s~ Has to return a float between ``0`` and ``1`` that indicates if a lexer wants to highlight this text. Used by ``guess_lexer``. If this method returns ``0`` it won't highlight it in any case, if it returns ``1`` highlighting with this lexer is guaranteed. The `LexerMeta` metaclass automatically wraps this function so that it works like a static method (no ``self`` or ``cls`` parameter) and the return value is automatically converted to `float`. If the return value is an object that is boolean `False` it's the same as if the return values was ``0.0``. N((ttext((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRkscsÐtˆtƒsòˆjdkryy5ˆjdƒ‰ˆjdƒrRˆtdƒ‰nWqïtk ruˆjdƒ‰qïXqòˆjdkrÝyddl}Wntk r·tdƒ‚nX|j ˆƒ}ˆj|d ƒ‰qòˆjˆjƒ‰nˆj d d ƒ‰ˆj d d ƒ‰ˆj r.ˆj ƒ‰nˆj rIˆj d ƒ‰nˆjd krmˆjˆjƒ‰nˆjr“ˆjd ƒ r“ˆd 7‰n‡‡fd†}|ƒ}|sÌt|ˆjˆƒ}n|S(s= Return an iterable of (tokentype, value) pairs generated from `text`. If `unfiltered` is set to `True`, the filtering mechanism is bypassed even if filters are defined. Also preprocess the text, i.e. expand tabs and strip it if wanted and applies registered filters. tguesssutf-8uR&tchardetiÿÿÿÿNskTo enable chardet encoding guessing, please install the chardet library from http://chardet.feedparser.org/R%s s s ic3s2x+ˆjˆƒD]\}}}||fVqWdS(N(tget_tokens_unprocessed(titttv(R-R4(s2/usr/lib/python2.7/site-packages/pygments/lexer.pytstreamer¡s(R2tunicodeR%tdecodet startswithtlentUnicodeDecodeErrorR6t ImportErrortdetecttreplaceR"tstripR!R$t expandtabsR#tendswithRR'(R-R4t unfilteredR6tencR;tstream((R-R4s2/usr/lib/python2.7/site-packages/pygments/lexer.pyt get_tokensys>       cCs t‚dS(s  Return an iterable of (tokentype, value) pairs. In subclasses, implement this method as a generator to maximize effectiveness. N(tNotImplementedError(R-R4((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR7©sN(RRR tNoneRtaliasest filenamestalias_filenamest mimetypesRt __metaclass__R/R1R,RR*RJR7(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR 's    0cBs#eZdZed„Zd„ZRS(s  This lexer takes two lexer as arguments. A root lexer and a language lexer. First everything is scanned using the language lexer, afterwards all ``Other`` tokens are lexed using the root lexer. The lexers from the ``template`` lexer package use this base lexer. cKs;|||_|||_||_tj||dS(N(t root_lexertlanguage_lexertneedleR R/(R-t _root_lexert_language_lexert_needleR(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR/¼s cCsÍd}g}g}x}|jj|ƒD]i\}}}||jkrx|rk|jt|ƒ|fƒg}n||7}q%|j|||fƒq%W|r´|jt|ƒ|fƒnt||jj|ƒƒS(Nt(RSR7RTR3R?t do_insertionsRR(R-R4tbufferedt insertionst lng_bufferR8R9R:((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR7Âs"  (RRR RR/R7(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR²s cBseZdZRS(sI Indicates that a state should include rules from another state. (RRR (((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRÙstcombinedcBs eZdZd„Zd„ZRS(s: Indicates a state combined from multiple states. cGstj||ƒS(N(ttupleR(Rtargs((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRåscGsdS(N((R-R_((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR/ès(RRR RR/(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR]às t _PseudoMatchcBsMeZdZd„Zdd„Zdd„Zdd„Zd„Zd„Z RS(s: A pseudo match object constructed from a string. cCs||_||_dS(N(t_textt_start(R-tstartR4((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR/òs cCs|jS(N(Rb(R-targ((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRcöscCs|jt|jƒS(N(RbR?Ra(R-Rd((s2/usr/lib/python2.7/site-packages/pygments/lexer.pytendùscCs|rtdƒ‚n|jS(Ns No such group(t IndexErrorRa(R-Rd((s2/usr/lib/python2.7/site-packages/pygments/lexer.pytgroupüscCs |jfS(N(Ra(R-((s2/usr/lib/python2.7/site-packages/pygments/lexer.pytgroupsscCsiS(N((R-((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyt groupdictsN( RRR R/RLRcReRgRhRi(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR`ís     csd‡fd†}|S(sL Callback that yields multiple actions for each group in the match. c3sxåtˆƒD]×\}}|dkr+q q t|ƒtkrw|j|dƒ}|rä|j|dƒ||fVqäq |r–|j|dƒ|_nxK||t|j|dƒ|j|dƒƒ|ƒD]}|rÌ|VqÌqÌWq W|r|jƒ|_ndS(Ni( t enumerateRLRRRgRctposR`Re(tlexertmatchtctxR8tactiontdatatitem(R_(s2/usr/lib/python2.7/site-packages/pygments/lexer.pytcallback s ! N(RL(R_Rr((R_s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRst_ThiscBseZdZRS(sX Special singleton used for indicating the caller class. Used by ``using``. (RRR (((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRs sc s–i‰dˆkrVˆjdƒ}t|ttfƒrC|ˆds (c3s‚ˆj|jƒˆˆ}|jƒ}x;|j|jƒˆD]!\}}}||||fVqAW|r~|jƒ|_ndS(N(RwR(RcR7RgReRk(RlRmRnRxRyR8R9R:(t_otherRzR{(s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRrMs  (N(tpopR2tlistR^RRL(R|R{RyRr((R|RzR{s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR(s     tRegexLexerMetacBsGeZdZd„Zd„Zd„Zd„Zdd„Zd„Z RS(sw Metaclass for RegexLexer, creates the self._tokens attribute from self.tokens on the first instantiation. cCstj||ƒjS(sBPreprocess the regular expression component of a token definition.(tretcompileRm(Rtregextrflags((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyt_process_regex`scCs5t|ƒtks1t|ƒs1td|fƒ‚|S(s5Preprocess the token component of a token definition.s2token type must be simple type or callable, not %r(RRtcallabletAssertionError(Rttoken((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyt_process_tokends!cCskt|tƒrz|dkrdS||kr2|fS|dkrB|S|d dkrat|dƒ Stsgtd|ƒ‚nít|tƒrd|j}|jd7_g}xF|D]>}||ksÔtd |ƒ‚|j|j|||ƒƒq²W|||<|fSt|t ƒrQx6|D].}||ks|d kstd |ƒ‚qW|Stsgtd |ƒ‚d S(s=Preprocess the state transition action of a token definition.s#popiÿÿÿÿs#pushis#pop:sunknown new state %rs_tmp_%discircular state ref %rsunknown new state sunknown new state def %rN(s#pops#push( R2tstrtintR*R†R]t_tmpnametextendt_process_stateR^(Rt new_statet unprocessedt processedt tmp_statetitokenstistate((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyt_process_new_statejs6        c Cs¨t|ƒtks"td|ƒ‚|ddksBtd|ƒ‚||krV||Sg}||<|j}x4||D](}t|tƒrÑ||ks©td|ƒ‚|j|j||t|ƒƒƒqxnt|ƒtksótd|ƒ‚y|j |d|ƒ}Wn3t k rB}t d|d|||fƒ‚nX|j |dƒ} t |ƒd krqd } n|j|d ||ƒ} |j|| | fƒqxW|S( s%Preprocess a single state definition.swrong state name %rit#sinvalid state name %rscircular state reference %rswrong rule def %rs+uncompilable regex %r in state %r of %r: %siiN(RR‰R†tflagsR2RRŒRR^R„t Exceptiont ValueErrorRˆR?RLR”R3( RRRRtttokensRƒttdeftrexterrR‡RŽ((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRs2"   "   cCsRi}|j|<|p!|j|}x'|jƒD]}|j|||ƒq1W|S(s-Preprocess a dictionary of token definitions.(t _all_tokensR™tkeysR(RRt tokendefsRRt((s2/usr/lib/python2.7/site-packages/pygments/lexer.pytprocess_tokendef°s cOsjt|dƒsWi|_d|_t|dƒr<|jr<qW|jd|jƒ|_ntj|||ŽS(s:Instantiate cls after preprocessing its token definitions.t_tokensittoken_variantsRX( thasattrRR‹R¢R R™R¡Rt__call__(RR_tkwds((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR¤¸s  N( RRR R„RˆR”RRLR R¤(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRZs   # # cBs/eZdZeZejZiZdd„Z RS(s± Base for simple stateful regular expression-based lexers. Simplifies the lexing process so that you need only provide a list of states and regular expressions. Rvc csd}|j}t|ƒ}||d}xÕxÏ|D]Q\}}} |||ƒ} | r3t|ƒtkr€||| jƒfVnx||| ƒD] } | VqW| jƒ}| d k r€t| tƒr x¦| D]L} | dkrì|j ƒqÍ| dkr |j |dƒqÍ|j | ƒqÍWnOt| t ƒr9|| 3n6| dkrY|j |dƒnt sot d| ƒ‚||d}nPq3q3Wya||dkrÌ|d7}dg}|d}|td fVw,n|t||fV|d7}Wq,tk rýPq,Xq,d S( s} Split ``text`` into (tokentype, text) pairs. ``stack`` is the inital stack (default: ``['root']``) iiÿÿÿÿs#pops#pushswrong state def: %rs iRvu N(R¡R~RRRgReRLR2R^R}R3RŠR*R†RRRf( R-R4RuRkRŸt statestackt statetokenstrexmatchRoRŽtmRqRt((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR7åsN               (sroot( RRR RRQR€t MULTILINER–R™R7(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR Æs  cBs&eZdZddd„Zd„ZRS(s9 A helper object that holds lexer position data. cCs=||_||_|p!t|ƒ|_|p3dg|_dS(NRv(R4RkR?ReRu(R-R4RkRuRe((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR/ s  cCsd|j|j|jfS(NsLexerContext(%r, %r, %r)(R4RkRu(R-((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR1&sN(RRR RLR/R1(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRscBseZdZddd„ZRS(sE A RegexLexer that uses a context object to store its state. c csG|j}|s+t|dƒ}|d}n |}||jd}|j}xõxï|D]B\}}}|||j|jƒ} | rUt|ƒtkr½|j|| jƒfV| jƒ|_n<x||| |ƒD] } | VqÐW|sù||jd}n|d k r“t |t ƒr'|jj |ƒnXt |t ƒrC|j|3n<|dkri|jj|jdƒntstd|ƒ‚||jd}nPqUqUWy|j|jkr´Pn||jdkr|jd7_dg|_|d}|jtdfVwNn|jt||jfV|jd7_WqNtk r?PqNXqNd S( s Split ``text`` into (tokentype, text) pairs. If ``context`` is given, use this lexer context instead. iRviÿÿÿÿs#pushswrong state def: %rs iu N(R¡RRuR4RkReRRRgRLR2R^RŒRŠR3R*R†RRRf( R-R4tcontextRŸRnR§R¨RoRŽR©Rq((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR70sR          N(RRR RLR7(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR +sccsÿt|ƒ}y|jƒ\}}Wn(tk rLx|D] }|Vq9WdSXd}t}x|D]\}}} |dkr„|}nd} xÂ|rN|t| ƒ|krN| | ||!} ||| fV|t| ƒ7}x5|D]-\} } }|| |fV|t|ƒ7}qßW||} y|jƒ\}}Wqtk rJt}PqXqW||| | fV|t| ƒ| 7}q`Wx|rú|p‹d}x5|D]-\}}} ||| fV|t| ƒ7}q•Wy|jƒ\}}Wq|tk röt}Pq|Xq|WdS(sg Helper for lexers which must combine the results of several sublexers. ``insertions`` is a list of ``(index, itokens)`` pairs. Each ``itokens`` iterable should be inserted at position ``index`` into the token stream given by the ``tokens`` argument. The result is a combined token stream. TODO: clean up the code here. Ni(titertnextt StopIterationRLR)R?R*(R[R™tindexR’RqtrealpostinsleftR8R9R:toldittmpvaltit_indextit_tokentit_valuetp((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRYisL            ('R R€tpygments.filterRRtpygments.filtersRtpygments.tokenRRRRt pygments.utilRRR R t__all__t staticmethodt_default_analyseRRtobjectR RR‰RR^R]R`RRsRRRR RR RY(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyt s. ""  ‹'    2lU>