'[f?dZddlZddlmZddlmZddlTddlmZddl m Z ddl m Z Gd d eZ Gd d eZdS) z Corpus reader for corpora whose documents are xml files. (note -- not named 'xml' to avoid conflicting w/ standard xml package) N) ElementTree) CorpusReader)*)SeekableUnicodeStreamReader)ElementWrapper)WordPunctTokenizerc*eZdZdZddZddZddZdS) XMLCorpusReadera Corpus reader for corpora whose documents are xml files. Note that the ``XMLCorpusReader`` constructor does not take an ``encoding`` argument, because the unicode encoding is specified by the XML files themselves. See the XML specs for more info. Fc@||_tj|||dSN) _wrap_etreer__init__)selfrootfileids wrap_etrees N/var/www/piapp/venv/lib/python3.11/site-packages/nltk/corpus/reader/xmldocs.pyrzXMLCorpusReader.__init__!s%%dD'22222Nc|%t|jdkr |jd}t|tst d||5}tj| }dddn #1swxYwY|j rt|}|S)Nrz(Expected a single file identifier string) len_fileids isinstancestr TypeErrorabspathopenrparsegetrootr r)rfileidfpelts rxmlzXMLCorpusReader.xml%s >c$-00A55]1%F&#&& HFGG G \\& ! ! & & ( ( 2B#B''//11C 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2   & %%C s3'B&&B*-B*c||}||}t} |}n#|}YnxYwg}|D]_}|j}|Tt |tr||}| |} | | `|S)aE Returns all of the words and punctuation symbols in the specified file that were in text nodes -- ie, tags are ignored. Like the xml() method, fileid can only specify one file. :return: the given file's text nodes as a list of words and punctuation symbols :rtype: list(str) ) r#encodingr getiteratoritertextrbytesdecodetokenizeextend) rr r"r%word_tokenizeriteratoroutnoder(tokss rwordszXMLCorpusReader.words4shhv==((+-- "((HH "xxzzHHH ! !D9DdE**1;;x00D%..t44 4    s AA')Fr )__name__ __module__ __qualname____doc__rr#r2rrr r sZ3333    rr ceZdZdZdZdZd dZdZdZe de j e j zZ e d Ze d e j e j zZd Zdd ZdS) XMLCorpusViewam A corpus view that selects out specified elements from an XML file, and provides a flat list-like interface for accessing them. (Note: ``XMLCorpusView`` is not used by ``XMLCorpusReader`` itself, but may be used by subclasses of ``XMLCorpusReader``.) Every XML corpus view has a "tag specification", indicating what XML elements should be included in the view; and each (non-nested) element that matches this specification corresponds to one item in the view. Tag specifications are regular expressions over tag paths, where a tag path is a list of element tag names, separated by '/', indicating the ancestry of the element. Some examples: - ``'foo'``: A top-level element whose tag is ``foo``. - ``'foo/bar'``: An element whose tag is ``bar`` and whose parent is a top-level element whose tag is ``foo``. - ``'.*/foo'``: An element whose tag is ``foo``, appearing anywhere in the xml tree. - ``'.*/(foo|bar)'``: An wlement whose tag is ``foo`` or ``bar``, appearing anywhere in the xml tree. The view items are generated from the selected XML elements via the method ``handle_elt()``. By default, this method returns the element as-is (i.e., as an ElementTree object); but it can be overridden, either via subclassing or via the ``elt_handler`` constructor parameter. FiNc|r||_t|dz|_ ddi|_ ||}t |||dS)aW Create a new corpus view based on a specified XML file. Note that the ``XMLCorpusView`` constructor does not take an ``encoding`` argument, because the unicode encoding is specified by the XML files themselves. :type tagspec: str :param tagspec: A tag specification, indicating what XML elements should be included in the view. Each non-nested element that matches this specification corresponds to one item in the view. :param elt_handler: A function used to transform each element to a value for the view. If no handler is specified, then ``self.handle_elt()`` is called, which returns the element as an ElementTree object. The signature of elt_handler is:: elt_handler(elt, tagspec) -> value z\Zrr7)r%N) handle_eltrecompile_tagspec _tag_context_detect_encodingStreamBackedCorpusViewr)rr tagspec elt_handlerr%s rrzXMLCorpusView.__init__ust*  *)DO 7U?33 9G  ((00''fx'HHHHHrct|trW |}|}|nU#|wxYwt|d5}|}dddn #1swxYwY|t jrdS|t jrdS|t j rdS|t j rdS|t j rdSt d|}|r'|dSt d |}|r'|dSdS) Nrbz utf-16-bez utf-16-lez utf-32-bez utf-32-lezutf-8s!\s*<\?xml\b.*\bencoding="([^"]+)"rs!\s*<\?xml\b.*\bencoding='([^']+)')r PathPointerrreadlineclose startswithcodecs BOM_UTF16_BE BOM_UTF16_LE BOM_UTF32_BE BOM_UTF32_LEBOM_UTF8r<matchgroupr*)rr infilesms rr@zXMLCorpusView._detect_encodings fk * * & OO%%  fd## &vOO%% & & & & & & & & & & & & & & & <<+ , , ; <<+ , , ; <<+ , , ; <<+ , , ; << ( ( 7 HH:A > >  '771::$$&& & HH:A > >  '771::$$&& &ws(AA*=BB"%B"c|S)a Convert an element into an appropriate value for inclusion in the view. Unless overridden by a subclass or by the ``elt_handler`` constructor argument, this method simply returns ``elt``. :return: The view value corresponding to ``elt``. :type elt: ElementTree :param elt: The element that should be converted. :type context: str :param context: A string composed of element tags separated by forward slashes, indicating the XML context of the given element. For example, the string ``'foo/bar/baz'`` indicates that the element is a ``baz`` element whose parent is a ``bar`` element and whose grandparent is a top-level ``foo`` element. r7)rr"contexts rr;zXMLCorpusView.handle_elts ( ra; [^<]* ( (() | # comment () | # doctype decl (<[^!>][^>]*>)) # tag or PI [^<]*)* \Zz<\s*(?:/\s*)?([^\s>]+)a6 # Include these so we can skip them: (?P )| (?P )| (?P <\?.*?\?> )| (?P ]*(\[[^\]]*])?\s*>)| # These are the ones we actually care about: (?P <\s*[^>/\?!\s][^>]*/\s*> )| (?P <\s*[^>/\?!\s][^>]*> )| (?P <\s*/[^>/\?!\s][^>]*> )c~d}t|tr|} ||j}||z }|j|r|Std| ddkrd|t|td| z z }td|z|std| d}|dkr|j|d |rqt|tr+||||n'|t||z d |d |S) a{ Read a string from the given stream that does not contain any un-closed tags. In particular, this function first reads a block from the stream of size ``self._BLOCK_SIZE``. It then checks if that block contains an un-closed tag. If it does, then this function either backtracks to the last '<', or reads another block. Tz[<>]r>zUnexpected ">" near char %sz&Unexpected end of file: tag not closed!DEEE K !IJJJ !)s 3 3  1$$%++H5G6G5G,HII8!&*EFFM H---001BCCCC c(mm6G&G$H!LLL#$6%6$677? 8rc ||j}|jt|j|}|Jg}d}d}d}|gks|t |tr|} ||} | s|ntd|j | D]} |j rPtdd|dd| | dr|j| d} || |Qt(|d|r#| }t-|}| d r|j| d} |std | z| |d krtd |d d | d|b|t-|krO|| || z }||d|fdx}}d}|"| dr|j| d} |vt(|d|dz| zrB|| d|dz| zf||gkr|| |dz }d}n|j rtdt |tr+|| ||n'|t-| |z d|d|dz }dx}}d}|gk ||} | |jvr!t7||j| ksJnt7||j| <fd|DS)z Read from ``stream`` until we find at least one element that matches ``tagspec``, and return the result of applying ``elt_handler`` to each element found. NrXzUnexpected end of filez {:>25} {}/i START_TAGrEND_TAGzUnmatched tag zUnmatched tag ...z,XMLCorpusView.read_block..s[   g K&szz';N'O'OPP     r)r>r;listr?getr[rrrkra _XML_PIECEfinditer_DEBUGprintformatjoinrQ _XML_TAG_NAMErPappendr<startrr`poprcrdtuple)rrerBrCrVelts elt_start elt_depthelt_textrg xml_fragmentpiecenameris ` r read_blockzXMLCorpusView.read_block"s ?mG  /Kt(,,V[[]];;<<"""  bjjI1&"=>> )!;;==226::L  ?$$%=>>>11,??" Y" Y;V+,,SXXg->->stt-DekkmmTTUUU;;{++Y-33EKKMMBBHHKKDNN4((( (88GSXXg->->??5(- I(+G I[[++Y-33EKKMMBBHHKKD"G()>)EFFFwr{**()U72;)U)Ud)U)U)UVVV ,c'll1J1J LUYY[[1H$II Xsxx/@/@$ABBB044 I#%KKMMMM[[11Y-33EKKMMBBHHKKD (88GSXXg->->-Dt-KLLY KK8I8IC8ORV8V(WXXX$2:: YZZ 88H !II{86777!&*EFFI H---00;;;; c,&7&7)&C$DaHHH%o A o6G,00I !HObjjI1Tkkmm $# # #>>T%6s%;;;;;;%*7^^D c "    #'     rr )NN)r3r4r5r6r}r]rr@r;r<r=DOTALLVERBOSEr^rr{rkrr7rrr9r9Qs<FK"I"I"I"IH:0JJ  BJ  MJJ899M  E BJ  J,8,8,8bk k k k k k rr9)r6rJ xml.etreernltk.corpus.reader.apirnltk.corpus.reader.util nltk.datarnltk.internalsr nltk.tokenizerr rAr9r7rrrs  !!!!!!//////%%%%111111)))))),,,,,,66666l666r| | | | | *| | | | | r