zIfpKddlZddlZddlZddlmZddlmZmZmZm Z m Z ddl m Z ddl mZddlmZdZGdd e ZGd d eZGd d eZGddeZdS)N)PIPE) _java_options config_java find_jar_iterfind_jars_within_pathjava)ParserIDependencyGraph)Treez2https://nlp.stanford.edu/software/lex-parser.shtmlcreZdZdZdZdZdZdZdZ dd Z d Z dd Z ddZ ddZ ddZddZddZdS)GenericStanfordParserz Interface to the Stanford Parserz+stanford-parser-(\d+)(\.(\d+))+-models\.jarzstanford-parser\.jarz3edu.stanford.nlp.parser.lexparser.LexicalizedParserFN4edu/stanford/nlp/models/lexparser/englishPCFG.ser.gzutf8-mx4gc tt|j|ddt|dd}tt|j|ddt|dd} t j|d } t| gt| z|_ ||_ ||_ ||_ ||_dS) N)STANFORD_PARSERSTANFORD_CORENLPT)env_vars searchpathurlverboseis_regexc@tj|SNospathdirname model_paths G/var/www/piapp/venv/lib/python3.11/site-packages/nltk/parse/stanford.pyz0GenericStanfordParser.__init__..<27??:#>#>)key)STANFORD_MODELSrc@tj|Srrr"s r$r%z0GenericStanfordParser.__init__..Ir&r'r)maxr_JAR _stanford_url_MODEL_JAR_PATTERNrr splittupler _classpathr# _encodingcorenlp_options java_options) self path_to_jarpath_to_models_jarr#encodingrr4r3 stanford_jar model_jar stanford_dirs r$__init__zGenericStanfordParser.__init__&s  @!   ?>     '"@!   ?>    w}}\2215  .CL.Q.Q QRR$!.(r'c g}g}g}d}|dD]}|dkr|r'|t|g}d}1|jr@||d|g}d}x|t|d|gg}||d}t|S)NFr T) splitlinesappenditer_DOUBLE_SPACED_OUTPUT _make_treejoin)r5output_res cur_lines cur_treesblanklines r$_parse_trees_outputz)GenericStanfordParser._parse_trees_outputWs  &&u--  Drzz #JJtI/// "I!EE/#$$T__TYYy5I5I%J%JKKK "I EEJJtT__TYYy5I5I%J%J$KLLMMM "II  &&&Cyyr'c |jd|jddd|jdddg }|||dd |D|S) a Use StanfordParser to parse multiple sentences. Takes multiple sentences as a list where each sentence is a list of words. Each sentence will be automatically tagged with this StanfordParser instance's tagger. If whitespaces exists inside a token, then the token will be treated as separate tokens. :param sentences: Input sentences to parse :type sentences: list(list(str)) :rtype: iter(iter(Tree)) -model -sentencesnewline -outputFormat -tokenizedz-escaperz-edu.stanford.nlp.process.PTBEscapingProcessorr>c3@K|]}d|VdS) NrD).0sentences r$ z4GenericStanfordParser.parse_sents..s.LLhsxx11LLLLLLr' _MAIN_CLASSr#_OUTPUT_FORMATrK_executerDr5 sentencesrcmds r$ parse_sentsz!GenericStanfordParser.parse_sentsns~    O        ;  '' MMTYYLL)LLLLLg     r'cJt||g|S)a& Use StanfordParser to parse a sentence. Takes a sentence as a string; before parsing, it will be automatically tokenized and tagged by the Stanford Parser. :param sentence: Input sentence to parse :type sentence: str :rtype: iter(Tree) )nextraw_parse_sentsr5rVrs r$ raw_parsezGenericStanfordParser.raw_parses$D(((W==>>>r'c|jd|jddd|jg}|||d||S)aI Use StanfordParser to parse multiple sentences. Takes multiple sentences as a list of strings. Each sentence will be automatically tokenized and tagged by the Stanford Parser. :param sentences: Input sentences to parse :type sentences: list(str) :rtype: iter(iter(Tree)) rMrNrOrPr>rXr\s r$rbz%GenericStanfordParser.raw_parse_sentssa    O      '' MM#tyy33W = =   r'cJt||g|S)a0 Use StanfordParser to parse a sentence. Takes a sentence as a list of (word, tag) tuples; the sentence must have already been tokenized and tagged. :param sentence: Input sentence to parse :type sentence: list(tuple(str, str)) :rtype: iter(Tree) )ratagged_parse_sentsrcs r$ tagged_parsez"GenericStanfordParser.tagged_parses$D++XJ@@AAAr'cd|jd|jddd|jdddd d d g}|||d fd |D|S)ad Use StanfordParser to parse multiple sentences. Takes multiple sentences where each sentence is a list of (word, tag) tuples. The sentences must have already been tokenized and tagged. :param sentences: Input sentences to parse :type sentences: list(list(tuple(str, str))) :rtype: iter(iter(Tree)) /rMrNrOrPrQz -tagSeparatorz-tokenizerFactoryz,edu.stanford.nlp.process.WhitespaceTokenizerz-tokenizerMethodnewCoreLabelTokenizerFactoryr>c3ZK|]%}dfd|DV&dS)rSc3BK|]}|VdSrrT)rUtagged tag_separators r$rWzEGenericStanfordParser.tagged_parse_sents...s1OOF]//77OOOOOOr'NrT)rUrVros r$rWz;GenericStanfordParser.tagged_parse_sents..sX HHOOOOhOOOOOr'rX)r5r]rr^ros @r$rgz(GenericStanfordParser.tagged_parse_sentss    O          :  * "'' MM $-      r'c|j}|d|g|jr,||jdt }t |j|tj dd5}t|tr|r| |}| |||jr<|dt#||j|t&t&\}}n?||jt#||jt&t& \}}|d d }|d d }||}dddn #1swxYwYt1j|jt |d|S) Nz -encodingrS)optionsrwbF)modedeleter) classpathstdinstdoutstderr)rurwrxs  s)r2extendr3r/rDrrr4tempfileNamedTemporaryFile isinstancestrencodewriteflush _USE_STDINseekrr1rr@namereplacedecoderunlink) r5r^input_rr8default_options input_filerwrxs r$r[zGenericStanfordParser._executes> K*+++   5 JJt+1133 4 4 4((=11 D-w???? (d5 A A A -Z&#&& 18 1x00   V $ $ $       """!%"o$ """ :?+++!%4?4"""^^K66F^^K66F]]8,,F3 - - - - - - - - - - - - - - -6 */""" OU;;;; sDF==GG)NNrrFrrF)__name__ __module__ __qualname____doc__r.r,rYrrBr<rKr_rdrbrhrgr[rr'r$rrs**G "DGKJ!I/)/)/)/)b.    > ? ? ? ?    . B B B B% % % % N,,,,,,r'rc,eZdZdZdZfdZdZxZS)StanfordParsera >>> parser=StanfordParser( ... model_path="edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz" ... ) # doctest: +SKIP >>> list(parser.raw_parse("the quick brown fox jumps over the lazy dog")) # doctest: +NORMALIZE_WHITESPACE +SKIP [Tree('ROOT', [Tree('NP', [Tree('NP', [Tree('DT', ['the']), Tree('JJ', ['quick']), Tree('JJ', ['brown']), Tree('NN', ['fox'])]), Tree('NP', [Tree('NP', [Tree('NNS', ['jumps'])]), Tree('PP', [Tree('IN', ['over']), Tree('NP', [Tree('DT', ['the']), Tree('JJ', ['lazy']), Tree('NN', ['dog'])])])])])])] >>> sum([list(dep_graphs) for dep_graphs in parser.raw_parse_sents(( ... "the quick brown fox jumps over the lazy dog", ... "the quick grey wolf jumps over the lazy fox" ... ))], []) # doctest: +NORMALIZE_WHITESPACE +SKIP [Tree('ROOT', [Tree('NP', [Tree('NP', [Tree('DT', ['the']), Tree('JJ', ['quick']), Tree('JJ', ['brown']), Tree('NN', ['fox'])]), Tree('NP', [Tree('NP', [Tree('NNS', ['jumps'])]), Tree('PP', [Tree('IN', ['over']), Tree('NP', [Tree('DT', ['the']), Tree('JJ', ['lazy']), Tree('NN', ['dog'])])])])])]), Tree('ROOT', [Tree('NP', [Tree('NP', [Tree('DT', ['the']), Tree('JJ', ['quick']), Tree('JJ', ['grey']), Tree('NN', ['wolf'])]), Tree('NP', [Tree('NP', [Tree('NNS', ['jumps'])]), Tree('PP', [Tree('IN', ['over']), Tree('NP', [Tree('DT', ['the']), Tree('JJ', ['lazy']), Tree('NN', ['fox'])])])])])])] >>> sum([list(dep_graphs) for dep_graphs in parser.parse_sents(( ... "I 'm a dog".split(), ... "This is my friends ' cat ( the tabby )".split(), ... ))], []) # doctest: +NORMALIZE_WHITESPACE +SKIP [Tree('ROOT', [Tree('S', [Tree('NP', [Tree('PRP', ['I'])]), Tree('VP', [Tree('VBP', ["'m"]), Tree('NP', [Tree('DT', ['a']), Tree('NN', ['dog'])])])])]), Tree('ROOT', [Tree('S', [Tree('NP', [Tree('DT', ['This'])]), Tree('VP', [Tree('VBZ', ['is']), Tree('NP', [Tree('NP', [Tree('NP', [Tree('PRP$', ['my']), Tree('NNS', ['friends']), Tree('POS', ["'"])]), Tree('NN', ['cat'])]), Tree('PRN', [Tree('-LRB-', [Tree('', []), Tree('NP', [Tree('DT', ['the']), Tree('NN', ['tabby'])]), Tree('-RRB-', [])])])])])])])] >>> sum([list(dep_graphs) for dep_graphs in parser.tagged_parse_sents(( ... ( ... ("The", "DT"), ... ("quick", "JJ"), ... ("brown", "JJ"), ... ("fox", "NN"), ... ("jumped", "VBD"), ... ("over", "IN"), ... ("the", "DT"), ... ("lazy", "JJ"), ... ("dog", "NN"), ... (".", "."), ... ), ... ))],[]) # doctest: +NORMALIZE_WHITESPACE +SKIP [Tree('ROOT', [Tree('S', [Tree('NP', [Tree('DT', ['The']), Tree('JJ', ['quick']), Tree('JJ', ['brown']), Tree('NN', ['fox'])]), Tree('VP', [Tree('VBD', ['jumped']), Tree('PP', [Tree('IN', ['over']), Tree('NP', [Tree('DT', ['the']), Tree('JJ', ['lazy']), Tree('NN', ['dog'])])])]), Tree('.', ['.'])])])] penncrtjdtdtj|i|dS)NzcThe StanfordParser will be deprecated Please use nltk.parse.corenlp.CoreNLPParser instead. stacklevelwarningswarnDeprecationWarningsuperr<r5argskwargs __class__s r$r<zStanfordParser.__init__GsL  R       $)&)))))r'c*tj|Sr)r fromstringr5results r$rCzStanfordParser._make_treeQsv&&&r'rrrrrZr<rC __classcell__rs@r$rrsY00dN*****'''''''r'rc,eZdZdZdZfdZdZxZS)StanfordDependencyParsera >>> dep_parser=StanfordDependencyParser( ... model_path="edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz" ... ) # doctest: +SKIP >>> [parse.tree() for parse in dep_parser.raw_parse("The quick brown fox jumps over the lazy dog.")] # doctest: +NORMALIZE_WHITESPACE +SKIP [Tree('jumps', [Tree('fox', ['The', 'quick', 'brown']), Tree('dog', ['over', 'the', 'lazy'])])] >>> [list(parse.triples()) for parse in dep_parser.raw_parse("The quick brown fox jumps over the lazy dog.")] # doctest: +NORMALIZE_WHITESPACE +SKIP [[((u'jumps', u'VBZ'), u'nsubj', (u'fox', u'NN')), ((u'fox', u'NN'), u'det', (u'The', u'DT')), ((u'fox', u'NN'), u'amod', (u'quick', u'JJ')), ((u'fox', u'NN'), u'amod', (u'brown', u'JJ')), ((u'jumps', u'VBZ'), u'nmod', (u'dog', u'NN')), ((u'dog', u'NN'), u'case', (u'over', u'IN')), ((u'dog', u'NN'), u'det', (u'the', u'DT')), ((u'dog', u'NN'), u'amod', (u'lazy', u'JJ'))]] >>> sum([[parse.tree() for parse in dep_graphs] for dep_graphs in dep_parser.raw_parse_sents(( ... "The quick brown fox jumps over the lazy dog.", ... "The quick grey wolf jumps over the lazy fox." ... ))], []) # doctest: +NORMALIZE_WHITESPACE +SKIP [Tree('jumps', [Tree('fox', ['The', 'quick', 'brown']), Tree('dog', ['over', 'the', 'lazy'])]), Tree('jumps', [Tree('wolf', ['The', 'quick', 'grey']), Tree('fox', ['over', 'the', 'lazy'])])] >>> sum([[parse.tree() for parse in dep_graphs] for dep_graphs in dep_parser.parse_sents(( ... "I 'm a dog".split(), ... "This is my friends ' cat ( the tabby )".split(), ... ))], []) # doctest: +NORMALIZE_WHITESPACE +SKIP [Tree('dog', ['I', "'m", 'a']), Tree('cat', ['This', 'is', Tree('friends', ['my', "'"]), Tree('tabby', ['the'])])] >>> sum([[list(parse.triples()) for parse in dep_graphs] for dep_graphs in dep_parser.tagged_parse_sents(( ... ( ... ("The", "DT"), ... ("quick", "JJ"), ... ("brown", "JJ"), ... ("fox", "NN"), ... ("jumped", "VBD"), ... ("over", "IN"), ... ("the", "DT"), ... ("lazy", "JJ"), ... ("dog", "NN"), ... (".", "."), ... ), ... ))],[]) # doctest: +NORMALIZE_WHITESPACE +SKIP [[((u'jumped', u'VBD'), u'nsubj', (u'fox', u'NN')), ((u'fox', u'NN'), u'det', (u'The', u'DT')), ((u'fox', u'NN'), u'amod', (u'quick', u'JJ')), ((u'fox', u'NN'), u'amod', (u'brown', u'JJ')), ((u'jumped', u'VBD'), u'nmod', (u'dog', u'NN')), ((u'dog', u'NN'), u'case', (u'over', u'IN')), ((u'dog', u'NN'), u'det', (u'the', u'DT')), ((u'dog', u'NN'), u'amod', (u'lazy', u'JJ'))]] conll2007crtjdtdtj|i|dS)NzwThe StanfordDependencyParser will be deprecated Please use nltk.parse.corenlp.CoreNLPDependencyParser instead.rrrrs r$r<z!StanfordDependencyParser.__init__sL  \       $)&)))))r'c$t|dS)Nroottop_relation_labelr rs r$rCz#StanfordDependencyParser._make_treev&AAAAr'rrs@r$rrUs`..`!N*****BBBBBBBr'rcHeZdZdZdZdZdZdZdZdZ fdZ d d Z d Z xZ S) StanfordNeuralDependencyParserar >>> from nltk.parse.stanford import StanfordNeuralDependencyParser # doctest: +SKIP >>> dep_parser=StanfordNeuralDependencyParser(java_options='-mx4g')# doctest: +SKIP >>> [parse.tree() for parse in dep_parser.raw_parse("The quick brown fox jumps over the lazy dog.")] # doctest: +NORMALIZE_WHITESPACE +SKIP [Tree('jumps', [Tree('fox', ['The', 'quick', 'brown']), Tree('dog', ['over', 'the', 'lazy']), '.'])] >>> [list(parse.triples()) for parse in dep_parser.raw_parse("The quick brown fox jumps over the lazy dog.")] # doctest: +NORMALIZE_WHITESPACE +SKIP [[((u'jumps', u'VBZ'), u'nsubj', (u'fox', u'NN')), ((u'fox', u'NN'), u'det', (u'The', u'DT')), ((u'fox', u'NN'), u'amod', (u'quick', u'JJ')), ((u'fox', u'NN'), u'amod', (u'brown', u'JJ')), ((u'jumps', u'VBZ'), u'nmod', (u'dog', u'NN')), ((u'dog', u'NN'), u'case', (u'over', u'IN')), ((u'dog', u'NN'), u'det', (u'the', u'DT')), ((u'dog', u'NN'), u'amod', (u'lazy', u'JJ')), ((u'jumps', u'VBZ'), u'punct', (u'.', u'.'))]] >>> sum([[parse.tree() for parse in dep_graphs] for dep_graphs in dep_parser.raw_parse_sents(( ... "The quick brown fox jumps over the lazy dog.", ... "The quick grey wolf jumps over the lazy fox." ... ))], []) # doctest: +NORMALIZE_WHITESPACE +SKIP [Tree('jumps', [Tree('fox', ['The', 'quick', 'brown']), Tree('dog', ['over', 'the', 'lazy']), '.']), Tree('jumps', [Tree('wolf', ['The', 'quick', 'grey']), Tree('fox', ['over', 'the', 'lazy']), '.'])] >>> sum([[parse.tree() for parse in dep_graphs] for dep_graphs in dep_parser.parse_sents(( ... "I 'm a dog".split(), ... "This is my friends ' cat ( the tabby )".split(), ... ))], []) # doctest: +NORMALIZE_WHITESPACE +SKIP [Tree('dog', ['I', "'m", 'a']), Tree('cat', ['This', 'is', Tree('friends', ['my', "'"]), Tree('tabby', ['-LRB-', 'the', '-RRB-'])])] conllz)edu.stanford.nlp.pipeline.StanfordCoreNLPz%stanford-corenlp-(\d+)(\.(\d+))+\.jarz,stanford-corenlp-(\d+)(\.(\d+))+-models\.jarTctjdtdtj|i||xjdz c_dS)Nz}The StanfordNeuralDependencyParser will be deprecated Please use nltk.parse.corenlp.CoreNLPDependencyParser instead.rrz(-annotators tokenize,ssplit,pos,depparse)rrrrr<r3rs r$r<z'StanfordNeuralDependencyParser.__init__sa  \       $)&)))  JJr'Fc td)z Currently unimplemented because the neural dependency parser (and the StanfordCoreNLP pipeline class) doesn't support passing in pre- tagged tokens. zxtagged_parse[_sents] is not supported by StanfordNeuralDependencyParser; use parse[_sents] or raw_parse[_sents] instead.)NotImplementedError)r5r]rs r$rgz1StanfordNeuralDependencyParser.tagged_parse_sentss " :   r'c$t|dS)NROOTrr rs r$rCz)StanfordNeuralDependencyParser._make_treerr'r)rrrrrZrYr,r.rrBr<rgrCrrs@r$rrs>N=K 3DHJ  K K K K K     BBBBBBBr'r)rr{r subprocessrnltk.internalsrrrrrnltk.parse.apir nltk.parse.dependencygraphr nltk.treer r-rrrrrr'r$rs #"""""666666D sssssGsssl@'@'@'@'@'*@'@'@'F?B?B?B?B?B4?B?B?BD?B?B?B?B?B%:?B?B?B?B?Br'