ó =ä!Mc @sĆdZddlZddlmZmZddlmZddlmZm Z m Z m Z ddl m Z mZmZmZddd d d d d ddg Zed„ƒZdefd„ƒYZdefd„ƒYZd efd„ƒYZd efd„ƒYZdefd„ƒYZdefd„ƒYZd„Zdefd„ƒYZeƒZ d„Z!defd„ƒYZ"defd „ƒYZ#d efd!„ƒYZ$d e#fd"„ƒYZ%d#„Z&dS($s´ pygments.lexer ~~~~~~~~~~~~~~ Base lexer classes. :copyright: Copyright 2006-2010 by the Pygments team, see AUTHORS. :license: BSD, see LICENSE for details. i˙˙˙˙N(t apply_filterstFilter(tget_filter_by_name(tErrortTexttOthert _TokenType(t get_bool_optt get_int_optt get_list_opttmake_analysatortLexert RegexLexertExtendedRegexLexertDelegatingLexert LexerContexttincludetbygroupstusingtthiscCsdS(Ng((tx((s2/usr/lib/python2.7/site-packages/pygments/lexer.pytst LexerMetacBseZdZd„ZRS(s‚ This metaclass automagically converts ``analyse_text`` methods into static methods which always return float values. cCs9d|kr#t|dƒ|ds(R(t __class__R(R-((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyt__repr__\s   cKs5t|tƒs!t||}n|jj|ƒdS(s8 Add a new stream filter to this lexer. N(t isinstanceRRR'tappend(R-R.R(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR,cscCsdS(s~ Has to return a float between ``0`` and ``1`` that indicates if a lexer wants to highlight this text. Used by ``guess_lexer``. If this method returns ``0`` it won't highlight it in any case, if it returns ``1`` highlighting with this lexer is guaranteed. The `LexerMeta` metaclass automatically wraps this function so that it works like a static method (no ``self`` or ``cls`` parameter) and the return value is automatically converted to `float`. If the return value is an object that is boolean `False` it's the same as if the return values was ``0.0``. N((ttext((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRkscsĐtˆtƒsňˆjdkryy5ˆjdƒ‰ˆjdƒrRˆtdƒ‰nWqďtk ruˆjdƒ‰qďXqňˆjdkrÝyddl}Wntk rˇtdƒ‚nX|j ˆƒ}ˆj|d ƒ‰qňˆjˆjƒ‰nˆj d d ƒ‰ˆj d d ƒ‰ˆj r.ˆj ƒ‰nˆj rIˆj d ƒ‰nˆjd krmˆjˆjƒ‰nˆjr“ˆjd ƒ r“ˆd 7‰n‡‡fd†}|ƒ}|sĚt|ˆjˆƒ}n|S(s= Return an iterable of (tokentype, value) pairs generated from `text`. If `unfiltered` is set to `True`, the filtering mechanism is bypassed even if filters are defined. Also preprocess the text, i.e. expand tabs and strip it if wanted and applies registered filters. tguesssutf-8uďťżR&tchardeti˙˙˙˙NskTo enable chardet encoding guessing, please install the chardet library from http://chardet.feedparser.org/R%s s s ic3s2x+ˆjˆƒD]\}}}||fVqWdS(N(tget_tokens_unprocessed(titttv(R-R4(s2/usr/lib/python2.7/site-packages/pygments/lexer.pytstreamerĄs(R2tunicodeR%tdecodet startswithtlentUnicodeDecodeErrorR6t ImportErrortdetecttreplaceR"tstripR!R$t expandtabsR#tendswithRR'(R-R4t unfilteredR6tencR;tstream((R-R4s2/usr/lib/python2.7/site-packages/pygments/lexer.pyt get_tokensys>       cCs t‚dS(s  Return an iterable of (tokentype, value) pairs. In subclasses, implement this method as a generator to maximize effectiveness. N(tNotImplementedError(R-R4((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR7ŠsN(RRR tNoneRtaliasest filenamestalias_filenamest mimetypesRt __metaclass__R/R1R,RR*RJR7(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR 's    0cBs#eZdZed„Zd„ZRS(s  This lexer takes two lexer as arguments. A root lexer and a language lexer. First everything is scanned using the language lexer, afterwards all ``Other`` tokens are lexed using the root lexer. The lexers from the ``template`` lexer package use this base lexer. cKs;|||_|||_||_tj||dS(N(t root_lexertlanguage_lexertneedleR R/(R-t _root_lexert_language_lexert_needleR(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR/źs cCsÍd}g}g}x}|jj|ƒD]i\}}}||jkrx|rk|jt|ƒ|fƒg}n||7}q%|j|||fƒq%W|r´|jt|ƒ|fƒnt||jj|ƒƒS(Nt(RSR7RTR3R?t do_insertionsRR(R-R4tbufferedt insertionst lng_bufferR8R9R:((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR7Âs"  (RRR RR/R7(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR˛s cBseZdZRS(sI Indicates that a state should include rules from another state. (RRR (((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRŮstcombinedcBs eZdZd„Zd„ZRS(s: Indicates a state combined from multiple states. cGstj||ƒS(N(ttupleR(Rtargs((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRĺscGsdS(N((R-R_((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR/čs(RRR RR/(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR]ŕs t _PseudoMatchcBsMeZdZd„Zdd„Zdd„Zdd„Zd„Zd„Z RS(s: A pseudo match object constructed from a string. cCs||_||_dS(N(t_textt_start(R-tstartR4((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR/ňs cCs|jS(N(Rb(R-targ((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRcöscCs|jt|jƒS(N(RbR?Ra(R-Rd((s2/usr/lib/python2.7/site-packages/pygments/lexer.pytendůscCs|rtdƒ‚n|jS(Ns No such group(t IndexErrorRa(R-Rd((s2/usr/lib/python2.7/site-packages/pygments/lexer.pytgroupüscCs |jfS(N(Ra(R-((s2/usr/lib/python2.7/site-packages/pygments/lexer.pytgroupsscCsiS(N((R-((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyt groupdictsN( RRR R/RLRcReRgRhRi(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR`ís     csd‡fd†}|S(sL Callback that yields multiple actions for each group in the match. c3sxĺtˆƒD]×\}}|dkr+q q t|ƒtkrw|j|dƒ}|rä|j|dƒ||fVqäq |r–|j|dƒ|_nxK||t|j|dƒ|j|dƒƒ|ƒD]}|rĚ|VqĚqĚWq W|r|jƒ|_ndS(Ni( t enumerateRLRRRgRctposR`Re(tlexertmatchtctxR8tactiontdatatitem(R_(s2/usr/lib/python2.7/site-packages/pygments/lexer.pytcallback s ! N(RL(R_Rr((R_s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRst_ThiscBseZdZRS(sX Special singleton used for indicating the caller class. Used by ``using``. (RRR (((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRs sc s–i‰dˆkrVˆjdƒ}t|ttfƒrC|ˆds (c3s‚ˆj|jƒˆˆ}|jƒ}x;|j|jƒˆD]!\}}}||||fVqAW|r~|jƒ|_ndS(N(RwR(RcR7RgReRk(RlRmRnRxRyR8R9R:(t_otherRzR{(s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRrMs  (N(tpopR2tlistR^RRL(R|R{RyRr((R|RzR{s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR(s     tRegexLexerMetacBsGeZdZd„Zd„Zd„Zd„Zdd„Zd„Z RS(sw Metaclass for RegexLexer, creates the self._tokens attribute from self.tokens on the first instantiation. cCstj||ƒjS(sBPreprocess the regular expression component of a token definition.(tretcompileRm(Rtregextrflags((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyt_process_regex`scCs|S(s5Preprocess the token component of a token definition.((Rttoken((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyt_process_tokendscCsűt|tƒrd|dkrdS||kr2|fS|dkrB|S|d dkr÷t|dƒ Sn“t|tƒrÓd|j}|jd7_g}x*|D]"}|j|j|||ƒƒqœW|||<|fSt|tƒr÷x|D]}qéW|SdS( s=Preprocess the state transition action of a token definition.s#popi˙˙˙˙s#pushis#pop:s_tmp_%diN(R2tstrtintR]t_tmpnametextendt_process_stateR^(Rt new_statet unprocessedt processedt tmp_statetitokenstistate((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyt_process_new_statejs0       c Cs(||kr||Sg}||<|j}xö||D]ę}t|tƒrs|j|j||t|ƒƒƒq6ny|j|d|ƒ}Wn3tk rÂ}td|d|||fƒ‚nX|j |dƒ} t |ƒdkrńd} n|j |d||ƒ} |j || | fƒq6W|S(s%Preprocess a single state definition.is+uncompilable regex %r in state %r of %r: %siiN(tflagsR2RRŠR‹R‡R„t Exceptiont ValueErrorR†R?RLR’R3( RRRŽRtttokensRƒttdeftrexterrR…RŒ((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR‹s*     cCsRi}|j|<|p!|j|}x'|jƒD]}|j|||ƒq1W|S(s-Preprocess a dictionary of token definitions.(t _all_tokensR–tkeysR‹(RRt tokendefsRŽRt((s2/usr/lib/python2.7/site-packages/pygments/lexer.pytprocess_tokendef°s cOsjt|dƒsWi|_d|_t|dƒr<|jr<qW|jd|jƒ|_ntj|||ŽS(s:Instantiate cls after preprocessing its token definitions.t_tokensittoken_variantsRX( thasattrRšR‰RŸRR–RžRt__call__(RR_tkwds((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRĄ¸s  N( RRR R„R†R’R‹RLRRĄ(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRZs   # # cBs/eZdZeZejZiZdd„Z RS(są Base for simple stateful regular expression-based lexers. Simplifies the lexing process so that you need only provide a list of states and regular expressions. Rvc csďd}|j}t|ƒ}||d}xżxš|D];\}}} |||ƒ} | r3t|ƒtkr€||| jƒfVnx||| ƒD] } | VqW| jƒ}| d k rjt| tƒr x| D]L} | dkrě|j ƒqÍ| dkr |j |dƒqÍ|j | ƒqÍWn9t| t ƒr9|| 3n | dkrY|j |dƒn||d}nPq3q3Wya||dkrś|d7}dg}|d}|t dfVw,n|t ||fV|d7}Wq,tk rçPq,Xq,d S( s} Split ``text`` into (tokentype, text) pairs. ``stack`` is the inital stack (default: ``['root']``) ii˙˙˙˙s#pops#pushs iRvu N(RžR~RRRgReRLR2R^R}R3RˆRRRf( R-R4RuRkRœt statestackt statetokenstrexmatchRoRŒtmRqRt((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR7ĺsL               (sroot( RRR RRQR€t MULTILINER“R–R7(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR Ćs  cBs&eZdZddd„Zd„ZRS(s9 A helper object that holds lexer position data. cCs=||_||_|p!t|ƒ|_|p3dg|_dS(NRv(R4RkR?ReRu(R-R4RkRuRe((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR/ s  cCsd|j|j|jfS(NsLexerContext(%r, %r, %r)(R4RkRu(R-((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR1&sN(RRR RLR/R1(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRscBseZdZddd„ZRS(sE A RegexLexer that uses a context object to store its state. c cs1|j}|s+t|dƒ}|d}n |}||jd}|j}xßxŮ|D],\}}}|||j|jƒ} | rUt|ƒtkr˝|j|| jƒfV| jƒ|_n<x||| |ƒD] } | VqĐW|sů||jd}n|dk r}t |t ƒr'|jj |ƒnBt |t ƒrC|j|3n&|dkri|jj|jdƒn||jd}nPqUqUWy|j|jkržPn||jdkrí|jd7_dg|_|d}|jtdfVwNn|jt||jfV|jd7_WqNtk r)PqNXqNdS( s Split ``text`` into (tokentype, text) pairs. If ``context`` is given, use this lexer context instead. iRvi˙˙˙˙s#pushs iu N(RžRRuR4RkReRRRgRLR2R^RŠRˆR3RRRf( R-R4tcontextRœRnR¤RĽRoRŒRŚRq((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR70sP          N(RRR RLR7(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyR +sccs˙t|ƒ}y|jƒ\}}Wn(tk rLx|D] }|Vq9WdSXd}t}x|D]\}}} |dkr„|}nd} xÂ|rN|t| ƒ|krN| | ||!} ||| fV|t| ƒ7}x5|D]-\} } }|| |fV|t|ƒ7}qßW||} y|jƒ\}}Wqtk rJt}PqXqW||| | fV|t| ƒ| 7}q`Wx|rú|p‹d}x5|D]-\}}} ||| fV|t| ƒ7}q•Wy|jƒ\}}Wq|tk röt}Pq|Xq|WdS(sg Helper for lexers which must combine the results of several sublexers. ``insertions`` is a list of ``(index, itokens)`` pairs. Each ``itokens`` iterable should be inserted at position ``index`` into the token stream given by the ``tokens`` argument. The result is a combined token stream. TODO: clean up the code here. Ni(titertnextt StopIterationRLR)R?R*(R[R–tindexRRqtrealpostinsleftR8R9R:toldittmpvaltit_indextit_tokentit_valuetp((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyRYisL            ('R R€tpygments.filterRRtpygments.filtersRtpygments.tokenRRRRt pygments.utilRRR R t__all__t staticmethodt_default_analyseRRtobjectR RR‡RR^R]R`RRsRRRR RR RY(((s2/usr/lib/python2.7/site-packages/pygments/lexer.pyt s. ""  ‹'    2lU>