'[fXdZddlmZddlmZmZmZmZddlm Z m Z m Z m Z m Z mZddlmZmZmZmZmZmZmZmZmZmZmZmZddlmZddlmZGdd eZ Gd d eZ!Gd d eZ"GddeZ#GddeZ$GddeZ%GddeZ&GddeZ'GddeZ(ee$e%e#gZ)ee(e&e#gZ*ee(e'e#gZ+GddeZ,Gdde,Z-Gdde,Z.Gd d!e,Z/Gd"d#e!Z0d$Z1d%d%d%d%d&e,d'fd(Z2d)Z3e4d*krmdd+l5m6Z6e2e7e6d,Z8e,e8d-.Z9d/Z:e:;Ze>D]Z?e7e?d0Sd0S)1z` Extension of chart parsing implementation to handle grammars with feature structures as nodes. ) perf_counter)TYPE FeatStructfind_variablesunify)CFGFeatStructNonterminal Nonterminal Productionis_nonterminal is_terminal) BottomUpPredictCombineRuleBottomUpPredictRuleCachedTopDownPredictRuleChart ChartParserEdgeIEmptyPredictRuleFundamentalRule LeafInitRuleSingleEdgeFundamentalRuleTopDownInitRuleTreeEdge)logic)Treec`eZdZdZd dZedZd dZdZdZ d Z d Z fd Z xZ S)FeatureTreeEdgea A specialized tree edge that allows shared variable bindings between nonterminals on the left-hand side and right-hand side. Each ``FeatureTreeEdge`` contains a set of ``bindings``, i.e., a dictionary mapping from variables to values. If the edge is not complete, then these bindings are simply stored. However, if the edge is complete, then the constructor applies these bindings to every nonterminal in the edge whose symbol implements the interface ``SubstituteBindingsI``. rNc<i|t|kr)r'|}fd|D}itj||||_jt tf_dS)az Construct a new edge. If the edge is incomplete (i.e., if ``dotz,FeatureTreeEdge.__init__..Ns'<<<4::c8,,<<<) lenr!r__init__ _bindings_comparison_keytuplesorteditems)r%spanlhsrhsdotr$s` `r&r*zFeatureTreeEdge.__init__;s  H #c((??x?**S(++C<<<<<<<>CSCS>;K;KQR    r(cht|jd|f|j|j|jdz|S)a :return: A new ``FeatureTreeEdge`` formed from this edge. The new edge's dot position is increased by ``1``, and its end index will be replaced by ``new_end``. :rtype: FeatureTreeEdge :param new_end: The new end index. :type new_end: int :param bindings: Bindings for the new edge. :type bindings: dict r)r0r1r2r3r$)r_span_lhs_rhs_dot)r%new_endr$s r&move_dot_forwardz FeatureTreeEdge.move_dot_forwardcs?*Q-)   A      r(cZt|ts|S||SN) isinstancer substitute_bindings)r%ntr$s r&r!zFeatureTreeEdge._bindvs."344 I%%h///r(c\|||jSrA)r!nextsymr+r%s r&next_with_bindingsz"FeatureTreeEdge.next_with_bindings{s zz$,,..$.999r(c4|jS)zC Return a copy of this edge's bindings dictionary. )r+copyrGs r&r$zFeatureTreeEdge.bindings~s~""$$$r(ct|jgt|jzt|jzt|jztS)z` :return: The set of variables used by this edge. :rtype: set(Variable) fs_class)rr;listr<r+keysvaluesrrGs r& variableszFeatureTreeEdge.variablessu  YK49oo 4>&&(()) *4>((**++ ,     r(cB|r tSdddt |jDz}td|S)Nz{%s}z, c3 K|] }d|zV dS)z%s: %rNr )r"items r& z*FeatureTreeEdge.__str__..s7**$(4******r( ) is_completesuper__str__joinr.r+r/)r%r$ __class__s r&rYzFeatureTreeEdge.__str__s      577??$$ $ **,24>3G3G3I3I,J,J***!!Hggoo''44(44 4r()rNrA)__name__ __module__ __qualname____doc__r* staticmethodr7r?r!rHr$rQrY __classcell__)r[s@r&rr.s  WWWW6   \      &000 :::%%%    555555555r(rc4eZdZdZdZdZdZdZefdZ dS) FeatureChartzQ A Chart for feature grammars. :see: ``Chart`` for more information. c jikrtjSt}t |}|jvr|t fd|D}tj||gS)z Returns an iterator over the edges in this chart. See ``Chart.select`` for more information about the ``restrictions`` on the edges. c3NK|]}|V dSrA)_get_type_if_possible)r"key restrictionsr%s r&rUz&FeatureChart.select..sG  >AD & &|C'8 9 9      r()iter_edgesr.rOr-_indexes _add_indexget)r%rh restr_keysvalss`` r&selectzFeatureChart.selects 2   $$ $L--//00 :&&  T] * * OOJ ' ' '     EO     DM*-11$;;<<.UEH**+=74+=+=+?+?@@r(N)hasattrr ValueErrorrkrjr- setdefaultappend)r%rnrgr6rorvs` @r&rlzFeatureChart._add_indexs  > >C5#&& > !6!<=== >-/. j)K 4 4DLVD   T2 & & - -d 3 3 3 3  4 4r(cjD]J\}}tfd|D}||gKdS)zs A helper function for ``insert``, which registers the new edge with all existing indexes. c3nK|]/}t|V0dSrArsrus r&rUz6FeatureChart._register_with_indexes..rwr(N)rkr/r-rzr{)r%rvrnr6ros`` r&_register_with_indexesz#FeatureChart._register_with_indexess $(=#6#6#8#8 4 4 ZLVD   T2 & & - -d 3 3 3 3  4 4r(c\t|trt|vr |tS|S)z Helper function which returns the ``TYPE`` feature of the ``item``, if it exists, otherwise it returns the ``item`` itself )rBdictr)r%rTs r&rfz"FeatureChart._get_type_if_possibles- dD ! ! ddll: Kr(c#RK|d|jD]}t|trp|t |t krBt ||dr||d|Ed{VdS)Nr)startendT rename_vars)complete tree_class)rp _num_leavesrBrr1rrtrees)r%rrrvs r&parseszFeatureChart.parsessKKaT-=K>> R RDD/22 RXXZZ%t44488::u$???5 ::dTj:QQQQQQQQQ  R Rr(N) r\r]r^r_rprlr~rfrrr r(r&rcrcsy ===.444( 4 4 4(,RRRRRRr(rcceZdZdZdZdS)FeatureFundamentalRulea A specialized version of the fundamental rule that operates on nonterminals whose symbols are ``FeatStructNonterminal``s. Rather than simply comparing the nonterminals for equality, they are unified. Variable bindings from these unifications are collected and stored in the chart using a ``FeatureTreeEdge``. When a complete edge is generated, these bindings are applied to all nonterminals in the edge. The fundamental rule states that: - ``[A -> alpha \* B1 beta][i:j]`` - ``[B2 -> gamma \*][j:k]`` licenses the edge: - ``[A -> alpha B3 \* beta][i:j]`` assuming that B1 and B2 can be unified to generate B3. c#fK||kr=|r)|rt |t sdS|}|}t |t rt|sdS|t|tkrdS| }| | }t|||d}|dSn||krdS| }|||} || ||r| VdSdS)N used_varsFr)rr is_incompleterWrBrr1rFr rr$rename_variablesrQrr?insert_with_backpointer) r%chartgrammar left_edge right_edgefoundrFr$resultnew_edges r&applyzFeatureFundamentalRule.apply s MMOOz//11 1 1'')) 2&&(( 29o66 2 F  ##%% j/ 2 2 ,!'**   ""4(JNN,<,.>II  ( (9j I I NNNNN  r(Nr\r]r^r_rr r(r&rrs-*%%%%%r(rc2eZdZdZeZdZdZdS) FeatureSingleEdgeFundamentalRulez A specialized version of the completer / single edge fundamental rule that operates on nonterminals whose symbols are ``FeatStructNonterminal``. Rather than simply comparing the nonterminals for equality, they are unified. c#K|j}||d|D] }|||||Ed{V!dS)NF)rrWrF)_fundamental_rulerprr1r)r%rrrfrrs r&_apply_completez0FeatureSingleEdgeFundamentalRule._apply_complete?s  #  ""z~~?O?O&   G GIxxw :FF F F F F F F F F G Gr(c#K|j}||d|D] }|||||Ed{V!dS)NT)rrWr1)rrprrFr)r%rrrrrs r&_apply_incompletez2FeatureSingleEdgeFundamentalRule._apply_incompleteFs  #,,--//t9J9J9L9L'   G GJxxw :FF F F F F F F F F G Gr(N)r\r]r^r_rrrrr r(r&rr5sU/.00GGGGGGGGr(rceZdZdZdS)FeatureTopDownInitRulec#K||D]7}t|d}||dr|V8dS)Nr1rr ) productionsrrr7insert)r%rrprodrs r&rzFeatureTopDownInitRule.applyTsm''GMMOO'<<  D&66tQ??H||Hb))   r(Nr\r]r^rr r(r&rrSs#r(rceZdZdZdZdS)FeatureTopDownPredictRulea A specialized version of the (cached) top down predict rule that operates on nonterminals whose symbols are ``FeatStructNonterminal``. Rather than simply comparing the nonterminals for equality, they are unified. The top down expand rule states that: - ``[A -> alpha \* B1 beta][i:j]`` licenses the edge: - ``[B2 -> \* gamma][j:j]`` for each grammar production ``B2 -> gamma``, assuming that B1 and B2 can be unified. c#:K|rdS||}}t|sdS|}|j||fd}|d|ur |d|urdS||D]}|r\|d} t| r3|| krX| | |krrt| |drGt||} || dr| V||f|j||f<dS)N)NNrr9rTrr )rWrFrr rH_donermrr2r num_leavesleafrr1rr7r) r%rrrvrFr6nextsym_with_bindingsdonerfirstrs r&rzFeatureTopDownPredictRule.applyns       Fg&&  F !% 7 7 9 9z~~4e&66djjlltxxzz844 ||Htg.. -  r(Nrr r(r&rrs#r(rceZdZdZdS)FeatureEmptyPredictRulec#K|dD]^}t|dzD]7}t||}||dr|V8_dS)NT)emptyr9r )rrangerrr7r)r%rrrr6rs r&rzFeatureEmptyPredictRule.applys''d'33 # #Du//11A566 # #*::4GG<<"--#"NNN # # #r(Nrr r(r&rrs######r(rceZdZedefdZdS)FeatureChartParserc 4tj||f|||d|dS)N)strategytrace_chart_width chart_class)rr*)r%rrrr parser_argss r&r*zFeatureChartParser.__init__sH    /#         r(N)r\r]r^BU_LC_FEATURE_STRATEGYrcr*r r(r&rrs3(       r(rceZdZdZdS)FeatureTopDownChartParserc 8tj||tfi|dSrA)rr*TD_FEATURE_STRATEGYr%rrs r&r*z"FeatureTopDownChartParser.__init__&#D'3FVV+VVVVVr(Nr\r]r^r*r r(r&rr(WWWWWr(rceZdZdZdS)FeatureBottomUpChartParserc 8tj||tfi|dSrA)rr*BU_FEATURE_STRATEGYrs r&r*z#FeatureBottomUpChartParser.__init__rr(Nrr r(r&rrrr(rceZdZdZdS)$FeatureBottomUpLeftCornerChartParserc 8tj||tfi|dSrA)rr*rrs r&r*z-FeatureBottomUpLeftCornerChartParser.__init__s6# '1  5@     r(Nrr r(r&rrs#     r(rc0eZdZdZdZdZdZdZdZdS)InstantiateVarsCharta? A specialized chart that 'instantiates' variables whose names start with '@', by replacing them with unique new variables. In particular, whenever a complete edge is added to the chart, any variables in the edge's ``lhs`` whose names start with '@' will be replaced by unique new ``Variable``. c<t||dSrA)rcr*)r%tokenss r&r*zInstantiateVarsChart.__init__sdF+++++r(c`t|_t|dSrA)set _instantiatedrc initializerGs r&rzInstantiateVarsChart.initializes( UU%%%%%r(cz||jvrdS||t|||S)NF)rinstantiate_edgercr)r%rvchild_pointer_lists r&rzInstantiateVarsChart.insertsB 4% % %5 d###""4/ABBBr(c4t|tsdS|sdS||jvrdS||}|sdS|j||||_ dS)a^ If the edge is a ``FeatureTreeEdge``, and it is complete, then instantiate all variables whose names start with '@', by replacing them with unique new variables. Note that instantiation is done in-place, since the parsing algorithms might already hold a reference to the edge for future use. N) rBrrW _edge_to_cpls inst_varsraddr1rCr;)r%rvrs r&rz%InstantiateVarsChart.instantiate_edge%s$00  F!!  F 4% % % FNN4((   F t$$$HHJJ229== r(cbd|DS)Ncji|]0}|jd|tj1S)@)name startswithrunique_variable)r"vars r& z2InstantiateVarsChart.inst_vars..CsI   x""3'' &((   r()r1rQ)r%rvs r&rzInstantiateVarsChart.inst_varsBs7  xxzz++--    r(N) r\r]r^r_r*rrrrr r(r&rrsl,,,&&&CCC >>>:     r(rc8ddlm}|dS)NrFeatureGrammara  S -> NP VP PP -> Prep NP NP -> NP PP VP -> VP PP VP -> Verb NP VP -> Verb NP -> Det[pl=?x] Noun[pl=?x] NP -> "John" NP -> "I" Det -> "the" Det -> "my" Det[-pl] -> "a" Noun[-pl] -> "dog" Noun[-pl] -> "cookie" Verb -> "ate" Verb -> "saw" Prep -> "with" Prep -> "under" ) nltk.grammarr fromstringrs r& demo_grammarrOs1++++++  $ $   r(Tr9z$I saw John with a dog with my cookieczddl}ddl}tt} |rt| ttd|j|rtd||} t } || |} | | } t| | }|r!tdt | z z|r|D]}t|dStdt|dS)Nr*z Sentence:tracezTime: %sz Nr trees:) systimeprintrr\splitr chart_parserNrrr)) print_times print_grammar print_treesprint_sentencerparsersentrrrrtcprrtrees r&demor js:JJJKKK GGGnnG g  #v! k4   ZZ\\FA u % % %B NN6 " "E gmmoo.. / /E1 jLNNQ./000'  D $KKKK   k3u::&&&&&r(cXddl}|ddddl}|d}|ddd|ddddS)Nrzfor i in range(1): demo()z/tmp/profile.outrcum<)profilerunpstatsStats strip_dirs sort_stats print_stats)r rps r& run_profilersNNN KK+-?@@@MMM '((ALLNNfe,,88<<<LLNNeV,,88<<<<rsvCCCCCCCCCCCC                            j5j5j5j5j5hj5j5j5hMRMRMRMRMR5MRMRMRj;;;;;_;;;|GGGGG'@GGG<_5D5D5D5D5D 85D5D5Dz     !4   (B<#####.###LNN$$&& LNN  $$&& LNN%%''$$&&         &WWWWW 2WWW WWWWW!3WWW      +=   8 8 8 8 8 <8 8 8 @8  /''''D=== zDFFF EGGGd677G  G1 - - -B D ZZ\\F HHV  E d r(