yIfτUdZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z ddl mZmZmZmZddlmZddlmZmZmZddlmZdd lmZmZmZmZmZmZmZm Z m!Z!m"Z"ddl#Z#dd l#m$Z$dd l%m&Z&m'Z'm(Z(dd l)m*Z*m+Z+dd l,m-Z-m.Z.ddl/m0Z0m1Z1erddl2m3Z3dZ4ej5dZ6ede7Z8ede7Z9GddeZ:edee8fZ;ee8dfZede>fZ?GddZ@GddZAGddeZBe!de9e9fZCe4fdVd$ZDdWd(ZEdXd.ZFGd/d0eZGGd1d2ZH dYdZd=ZIejJGd>dZKd?d@iZLdAeMdB<d[dIZNGdJdKe'eHZOd\dNZPd]d^dQZQdYd_dTZReSdUkr eRdSdS)`aA similarities / code duplication command line tool and pylint checker. The algorithm is based on comparing the hash value of n successive lines of a file. First the files are read and any line that doesn't fulfill requirement are removed (comments, docstrings...) Those stripped lines are stored in the LineSet class which gives access to them. Then each index of the stripped lines collection is associated with the hash of n successive entries of the stripped lines starting at the current index (n is the minimum common lines option). The common hashes between both linesets are then looked for. If there are matches, then the match indices in both linesets are stored and associated with the corresponding couples (start line number/end line number) in both files. This association is then post-processed to handle the case of successive matches. For example if the minimum common lines setting is set to four, then the hashes are computed with four lines. If one of match indices couple (12, 34) is the successor of another one (11, 33) then it means that there are in fact five lines which are common. Once post-processed the values of association table are the result looked for, i.e. start and end lines numbers of common lines in both files. ) annotationsN) defaultdict)Callable GeneratorIterableSequence)getopt)BufferedIOBaseBufferedReaderBytesIO)chain) TYPE_CHECKINGAnyDictList NamedTupleNewTypeNoReturnTextIOTupleUnion)nodes) BaseCheckerBaseRawFileCheckertable_lines_from_stats)SectionTable)MessageDefinitionTupleOptions) LinterStatsdecoding_stream)PyLinterz.*\w+Index LineNumberc$eZdZUded<ded<dS) LineSpecifsr% line_numberstrtextN__name__ __module__ __qualname____annotations__K/var/www/piapp/venv/lib/python3.11/site-packages/pylint/checkers/similar.pyr'r'Qs% IIIIIr1r' LinesChunkSuccessiveLinesLimitsceZdZdZdZd d Zd S) CplSuccessiveLinesLimitszHolds a SuccessiveLinesLimits object for each checked file and counts the number of common lines between both stripped lines collections extracted from both files.  first_file second_fileeffective_cmn_lines_nbr8r4r9r:intreturnNonec0||_||_||_dSNr7)selfr8r9r:s r2__init__z!CplSuccessiveLinesLimits.__init__hs! %&&<###r1N)r8r4r9r4r:r;r<r=)r,r-r.__doc__ __slots__rAr0r1r2r6r6as:HI======r1r6LineSetStartCouplec>eZdZdZdZdd ZddZddZddZddZ dS)r3zlThe LinesChunk object computes and stores the hash of some consecutive stripped lines of a lineset. _fileid_index_hashfileidr)num_liner;lines Iterable[str]r<r=c|||_ t||_ td|D|_dS)Nc34K|]}t|VdSr?)hash).0lins r2 z&LinesChunk.__init__..s(99Cd3ii999999r1)rGr$rHsumrI)r@rJrKrLs r2rAzLinesChunk.__init__sD" Q"8__  99599999 11r1orboolcZt|tstS|j|jkSr?) isinstancer3NotImplementedrI)r@rUs r2__eq__zLinesChunk.__eq__s)!Z(( "! !zQW$$r1c|jSr?)rIr@s r2__hash__zLinesChunk.__hash__s zr1c8d|jd|jd|jdS)NzrFr\s r2__repr__zLinesChunk.__repr__s+ X4< X X4; X X$* X X X r1c6d|jd|jd|jS)NzLinesChunk object for file z, starting at line z Hash is rFr\s r2__str__zLinesChunk.__str__s7 $$, $ $4; $ $z $ $ r1N)rJr)rKr;rLrMr<r=)rUrr<rVr<r;r<r)) r,r-r.rBrCrArZr]r_rar0r1r2r3r3xs/I 2 2 2 2%%%%           r1cxeZdZdZdZddZedd Zedd Zej dd ZddZ dS)r4zA class to handle the numbering of begin and end of successive lines. :note: Only the end line number can be updated. _start_endstartr%endr<r=c"||_||_dSr?re)r@rhris r2rAzSuccessiveLinesLimits.__init__s"' # r1c|jSr?)rfr\s r2rhzSuccessiveLinesLimits.starts {r1c|jSr?rgr\s r2rizSuccessiveLinesLimits.ends yr1valuec||_dSr?rmr@rns r2rizSuccessiveLinesLimits.ends  r1r)c(d|jd|jdS)Nz>rer\s r2r_zSuccessiveLinesLimits.__repr__sE$+EE EEEEr1N)rhr%rir%r<r=)r<r%)rnr%r<r=rc) r,r-r.rBrCrApropertyrhrisetterr_r0r1r2r4r4s #I$$$$XX ZZFFFFFFr1cHeZdZUdZded<ded<ddZdd Zdd ZddZdS)rDzEIndices in both linesets that mark the beginning of successive lines.r$fst_lineset_indexsnd_lineset_indexr<r)c(d|jd|jdS)Nz.s$99Q!&999999r1c>g|]}t|dSr?)iter)rQirLs r2 z hash_lineset..s'FFFT%)__FFFr1)rhri)rlisttuplestripped_linesrange enumeratezipr%r( IndexErrorr$r4r3nameappend) rr hash2index index2lines shifted_linesr succ_linesstart_linenumberend_linenumberindexl_crLs @r2 hash_linesetrs^T""JK 99'"8999 9 9EFFFFe4D.E.EFFFM#C$788 & &J%g&r(B(NQR(RSSNNN Ta2"   Eu:z:::3u%%%% { ""s B!!,CC all_couplesCplIndexToCplLines_Tr=ct|D]}g}|td}||vr||jj||j_||jj||j_||xjdz c_|||td}||v|D](} | |#t$rY%wxYwdS)aRemoves all successive entries in the dictionary in argument. :param all_couples: collection that has to be cleaned up from successive entries. The keys are couples of indices that mark the beginning of common entries in both linesets. The values have two parts. The first one is the couple of starting and ending line numbers of common successive lines in the first file. The second part is the same for the second file. For example consider the following dict: >>> all_couples {(11, 34): ([5, 9], [27, 31]), (23, 79): ([15, 19], [45, 49]), (12, 35): ([6, 10], [28, 32])} There are two successive keys (11, 34) and (12, 35). It means there are two consecutive similar chunks of lines in both files. Thus remove last entry and update the last line numbers in the first entry >>> remove_successive(all_couples) >>> all_couples {(11, 34): ([5, 10], [27, 32]), (23, 79): ([15, 19], [45, 49])} rN) rkeysrr$r8rir9r:rpopKeyError)rcouple to_removetesttargets r2remove_successivers14 ((**++ a))k!!1>%((++D k!!   F ''''     s&C<< D D ls_1 stindex_1ls_2 stindex_2common_lines_nbcd|j|||zD}d|j|||zD}tdt||DS)a|Return the effective number of common lines between lineset1 and lineset2 filtered from non code lines. That is to say the number of common successive stripped lines except those that do not contain code (for example a line with only an ending parenthesis) :param ls_1: first lineset :param stindex_1: first lineset starting index :param ls_2: second lineset :param stindex_2: second lineset starting index :param common_lines_nb: number of common successive stripped lines before being filtered from non code lines :return: the number of common successive stripped lines that contain code cZg|](}t|j!|j)Sr0REGEX_FOR_LINES_WITH_CONTENTmatchr*rQlspecifs r2rz(filter_noncode_lines..D@  ' - -gl ; ; r1cZg|](}t|j!|j)Sr0rrs r2rz(filter_noncode_lines..Irr1c3(K|] \}}||kVdSr?r0)rQsline_1sline_2s r2rSz'filter_noncode_lines..Ns,XX&6gww'!XXXXXXr1)rrTr)rrrrr stripped_l1 stripped_l2s r2filter_noncode_linesr/s**9y?7R+RSK *9y?7R+RSK XX#k;:W:WXXX X XXr1cVeZdZUded<ded<ded<ded<ded<ded <ded <d S) Commonalityr; cmn_lines_nbrfst_lsetr%fst_file_start fst_file_endsnd_lsetsnd_file_start snd_file_endNr+r0r1r2rrQsfr1rcreZdZdZeddddfd&d Z d'd(dZd)dZd*dZd+dZ d,dZ d-dZ d.d Z d/d"Z d0d%Zd S)1Similarz-Finds copy-pasted lines of code in a project.F min_linesr;ignore_commentsrVignore_docstringsignore_importsignore_signaturesr<r=c t|tr|jj|_nt j|_||j_||j_||j_ ||j_ ||j_ g|_ dSr?) rXrlinterconfig namespaceargparse Namespacemin_similarity_linesrrrrlinesets)r@rrrrrs r2rAzSimilar.__init__^sq dK ( ( 2![/DNN%/11DN.7+)8&+<((6%+<(') r1Nstreamidr)stream STREAM_TYPESencoding str | Nonect|tr|tt||j}n|j} |}n#t $rg}YnwxYw|jt|||j j |j j |j j |j j t|dr |jjnddS)z)Append a file to search for similarities.Nrline_enabled_callback)rXr ValueErrorr! readlinesUnicodeDecodeErrorrrrrrrrrhasattrr_is_one_message_enabled)r@rrrrrLs r2 append_streamzSimilar.append_streamss fn - - )  '99CII(I IKKEE!   EEE   .0-0tX..DK77        s A AAcx|jjdkrdS||dS)z=Start looking for similarities and display results on stdout.rN)rr _display_sims _compute_simsr\s r2runz Similar.runs= > .! 3 3 F 4--//00000r1)list[tuple[int, set[LinesChunkLimits_T]]]ctt}|D]m}|j}|j}|j}|j}|j}|j}|j } ||} | D]} |||f| vs||| f| vrn| |||f||| fhng} | D]!\}} | D]}| ||f"| | | S)z'Compute similarities in appended files.)rr _iter_simsrrrrrrrritemssortreverse)r@ no_duplicates commonalitynumlineset1 start_line_1 end_line_1lineset2 start_line_2 end_line_2 duplicatecouplessims ensemblescplss r2rzSimilar._compute_simss^BMdBSBS ??,,  K*C"+H&5L$1J"+H&5L$1J%c*I$  lJ77BB G GG E G  !<<!<< ;=+1133 ) )NC! ) ) S$K(((( )   r1 similaritiescN||}t|dS)z(Display computed similarities on stdout.N)_get_similarity_reportprint)r@rreports r2rzSimilar._display_simss&,,\:: f r1c d}d}|D]\}}|d|dt|dz }t|}dx}x}} |D]\}}} |d|jd|d | d z }|rC|j|| D]3} || rd | dndz }4||t|d z zz }t d |jD} |d| d|d|dz| z ddz }|S)z"Create a report from similarities.r z similar lines in z files N==:[:z] z rc34K|]}t|VdSr?lenrQrs r2rSz1Similar._get_similarity_report..s($O$OgS\\$O$O$O$O$O$Or1z TOTAL lines=z duplicates=z percent=Y@z.2f)r sortedr _real_linesrstriprTr) r@rrduplicated_line_numbernumberr couples_lline_set start_lineend_linelinetotal_line_numbers r2rzSimilar._get_similarity_reports&'+ D DOFG K6KKS\\KKK KFwI/3 3H 3zH2; K K.*hJx}JJ JJXJJJJ Q$0H1DEQQDP5DKKMM5555DPFF "fI0B&C C " "!$$O$O$O$O$O!O!O R, R R0 R R-58II Q R R R   r1rrr"Generator[Commonality, None, None]c #Kt||jj\}t||jj\}}t}t|}t ||zfd}i} t |t jdD]} tj | || D]o} | d} | d} ttj || tj || |jj| t| | <pt| | D]\}}|j}|j}|j}t%|||jj|jj||jj|jj}t/|||||}||jjkr|VdS) atFind similarities in the two given linesets. This the core of the algorithm. The idea is to compute the hashes of a minimal number of successive lines of each lineset and then compare the hashes. Every match of such comparison is stored in a dict that links the couple of starting indices in both linesets to the couple of corresponding starting and ending lines in both files. Last regroups all successive couples in a bigger one. It allows to take into account common chunk of lines that have more than the minimal number of successive lines required. c |dS)Nrr0)mhash_to_index_1s r2z&Similar._find_common..s?1+=a+@r1)keyrHrr)r:)rrrrrrrN)rrr frozensetrr operator attrgetter itertoolsproductr6copyrDrrrwrxr:rr8rhrir9r)r@rrindex_to_lines_1hash_to_index_2index_to_lines_2hash_1hash_2 common_hashesrc_hashindices_in_linesetsindex_1index_2cml_stripped_lcmn_l start_index_1 start_index_2nb_common_linescom eff_cmn_nbrs @r2 _find_commonzSimilar._find_commons<&-9 dn9- - ))-9 dn9- - )))2/2F2F2H2H(I(I(1/2F2F2H2H(I(I.4 VO!@!@!@!@/ / / -/ ]0CH0M0MNNN  F'0'8')@((  #.a0-a0, "27";<< "27";<.As%UUU'WUUTUUUUr1Nr;)r@r=s r2combine_mapreduce_datazSimilar.combine_mapreduce_data<s VU-@UUU r1) rr;rrVrrVrrVrrVr<r=r?)rr)rrrrr<r=r<r=)r<r)rrr<r=)rrr<r))rrrrr<r)r<rr<r9)r=r>r<r=)r,r-r.rBDEFAULT_MIN_SIMILARITY_LINErArrrrrr6rr<rAr0r1r2rr[s775 %"'$"' *****,KO     B1111 %%%%N2IIIIV@@@@VVVVVVr1rrLrMrrVrrrr!Callable[[str, int], bool] | Nonelist[LineSpecifs]c N|s|r'tjd|}|r\i}|tjtjfD].}t|j|j p|jdzD]} d|| </|r2dfd g|} ttd | D} g} d } t|d D]O\} }| |d | s| }|r| s}|ds|dr|d d} |dd }n>|ds|dr|dd} |dd }| r|| rd } d}|r|| d}|rd}|r.|ddd }|r| | vrd}|r4| t'|t)| dz Q| S)aReturn tuples of line/line number/line type with leading/trailing white-space and any ignored code features removed. :param lines: a collection of lines :param ignore_comments: if true, any comment in the lines collection is removed from the result :param ignore_docstrings: if true, any line that is a docstring is removed from the result :param ignore_imports: if true, any line that is an import is removed from the result :param ignore_signatures: if true, any line that is part of a function signature is removed from the result :param line_enabled_callback: If called with "R0801" and a line number, a return value of False will disregard the line :return: the collection of line/line number/line type tuples rrT functionslist[nodes.NodeNG]tree nodes.NodeNGr<c |jD]z}t|tjtjfr||t|tjtjtjfr ||{|S)zeRecursively get all functions including nested in the classes from the tree. )bodyrXr FunctionDefAsyncFunctionDefrClassDef)rHrJnode_get_functionss r2rRz&stripped_lines.._get_functionsas  4 4dU%68N$OPP+$$T***^U%68NO4#N9d333 r1c3K|]:}t|j|jr|jdjn |jdzV;dS)rrN)rlinenorMtolineno)rQfuncs r2rSz!stripped_lines..wsf   /3yO ! ++dma>Or1N)rhR0801z"""z'''zr"""zr'''r#F#r)r*r()rHrIrJrKr<rI)astroidparsejoinnodes_of_classrImport ImportFromrrT end_linenosetr rstrip startswithendswithgetsplitrr'r%)rLrrrrrrJ import_linesrQrTrHsignature_lines strippedlines docstringrcurrent_line_is_importrRs @r2rrDs(-*-}RWWU^^,,, ''u7G(HII , ,D do.LPQ-QRR , ,'+ V$$ ,        &#N2t,,   !*     MI!%q111  ,5J5J V6 6 , zz||   $??5))$T__U-C-C$ $RaRI8DD__V,,$0G0G$ $QqS I8D ==++% $I  %1%5%5fe%D%D "%   1::c1%%a(..00D  ?!:!:D    :fqj3I3IJJJ    r1ceZdZdZ d d!dZd"dZd#dZd$dZd%dZd#dZ d&dZ e d'dZ e d(dZ dS))rzHolds and indexes all the lines of a single source file. Allows for correspondence between real lines of the source file and stripped ones, which are the real ones from which undesired patterns have been removed. FNrr)rL list[str]rrVrrrrrEr<r=cV||_||_t|||||||_dS)Nr)rrr_stripped_lines)r@rrLrrrrrs r2rAzLineSet.__init__sB  -     "7    r1cd|jdS)Nz rr\s r2razLineSet.__str__s+ty++++r1r;c*t|jSr?)r rr\s r2__len__zLineSet.__len__s4#$$$r1rr'c|j|Sr?ro)r@rs r2 __getitem__zLineSet.__getitem__s#E**r1rzc"|j|jkSr?rrr|s r2__lt__zLineSet.__lt__sy5:%%r1c t|Sr?)idr\s r2r]zLineSet.__hash__s $xxr1rcPt|tsdS|j|jkS)NF)rXr__dict__r|s r2rZzLineSet.__eq__s(%)) 5}..r1rFc|jSr?rvr\s r2rzLineSet.stripped_liness ##r1c|jSr?)rr\s r2 real_lineszLineSet.real_liness r1)FFFFN)rr)rLrmrrVrrVrrVrrVrrEr<r=rcrb)rr;r<r')rzrr<rVr)r<rF)r<rm)r,r-r.rBrArartrwryr]rZrtrrr0r1r2rrs!&"'$"'CG     *,,,,%%%%++++&&&&//// $$$X$   X   r1rW)zSimilar lines in %s files %szduplicate-codezIndicates that a set of similar lines has been detected among multiple file. This usually means that the code should be refactored to avoid this duplication.z!dict[str, MessageDefinitionTuple]MSGSsectrstatsr old_statsLinterStats | Nonecgd}|t||dz }|t|ddddS)z0Make a layout with some stats about duplication.)rnowprevious differenceduplicated_linesr#r)childrencolsrheaderscheadersN)rrr)rrrrLs r2report_similaritiesrsP 2 1 1E #E96H I IIEKKu1q1EEEFFFFFr1c eZdZUdZdZeZdeddddfdd d d d dfd d d d ddfdd d d ddfdd d d ddffZde d<dde ffZ d'dZ d(dZ d)dZd(d Zd*d"Zd+d%Zd&S),SimilarCheckerzChecks for similarities and duplicated code. This computation may be memory / CPU intensive, so you should disable it if you experience some problems. rzmin-similarity-linesr;zz%Minimum lines number of a similarity.)defaulttypemetavarhelpignore-commentsTynzz4Comments are removed from the similarity computationignore-docstringsz6Docstrings are removed from the similarity computationignore-importsz3Imports are removed from the similarity computationignore-signaturesz6Signatures are removed from the similarity computationroptionsRP0801 Duplicationrr"r<r=ctj||t||jjj|jjj|jjj|jjj|jjj dS)N)rrrrr) rrArrrrrrrr)r@rs r2rAzSimilarChecker.__init__4sv#D&111 k(= K.>"k0B;-<"k0B      r1cPg|_|jjdS)z=Init the checkers: reset linesets and statistics information.N)rrrreset_duplicated_linesr\s r2openzSimilarChecker.open?s&  0022222r1rQ nodes.Modulec|jjtjdtd|5}||jj||jddddS#1swxYwYdS)zProcess a module. the module's content is accessible via the stream object stream must implement the readlines method NzIn pylint 3.0 the current_name attribute of the linter object should be a string. If unknown it should be initialized as an empty string.) stacklevel)r current_namewarningswarnDeprecationWarningrr file_encoding)r@rQrs r2process_modulezSimilarChecker.process_moduleDs ; # + MN#     [[]] Uf   t{7AS T T T U U U U U U U U U U U U U U U U U Us'A11A58A5c td|jD}d}|jj}|D]\}}g}dx}x}} |D]*\}}} |d|jd|d| d+||r9|j|| D])} || *| dt|d |f ||t|d z zz }|xj t|z c_ |xjt!|o|d z|z z c_dS) zBCompute and display similarities on closing (i.e. end of parsing).c34K|]}t|VdSr?r r s r2rSz'SimilarChecker.close..Zs(>>WCLL>>>>>>r1rNrrr]rWr)argsrr )rTrrrrrrrrr add_messager r\nb_duplicated_linesr;percent_duplicated_linesfloat) r@total duplicatedrrrmsgrrrrs r2closezSimilarChecker.closeXs>> >>>>>  ! ..00 3 3LCC.2 2G 2j818 J J-X H HH HHXHHHIIII HHJJJ .#.z(/BC..DJJt{{}}----   WCLL$))C..+I  J J J #W!12 2JJ !!S__4!! &&%0T*u:Lu:T*U*UU&&&&r1r9c6t|S)zPassthru override.)rr<r\s r2r<zSimilarChecker.get_map_datams##D)))r1datar>cft|||dS)zReduces and recombines data into a format that we can report on. The partner function of get_map_data() Calls self.close() to actually calculate and report duplicate code. )r=N)rrAr)r@rrs r2reduce_map_datazSimilarChecker.reduce_map_dataqs. &&t&FFF r1Nrr"r<r=rB)rQrr<r=rC)rr"rr>r<r=)r,r-r.rBrrmsgsrDrr/rreportsrArrrr<rr0r1r2rrs{ D D #6"?     %N     %P     %M     %P    K.G....`-)<=?G     3333 UUUU(VVVV*****r1rrr"cJ|t|dSr?)register_checkerr)rs r2registerr|s$ N62233333r1statusrctdttdtj|dS)z'Display command line usage information.z*finds copy pasted blocks in a set of fileszUsage: symilar [-d|--duplicates min_duplicated_lines] [-i|--ignore-comments] [--ignore-docstrings] [--ignore-imports] [--ignore-signatures] file1...N)rsysexit)rs r2usagersJ 6777 GGG  `HVr1argvSequence[str] | NonecR|tjdd}d}gd}t}d}d}d}d}tt |||\}} |D]G\} } | dvrt | }| dvrt ,| dvrd }3| d vrd }:| d vrd }A| d vrd }H| st dt|||||} | D]A} t| d 5}| | |dddn #1swxYwYB| tj ddS)z%Standalone command line access point.Nrhdi)rz duplicates=rrrrF>-d --duplicates>-h--help>-i--ignore-commentsT>--ignore-docstrings>--ignore-imports>--ignore-signatureszutf-8)rr) rrrDr rr;rrrrrr)rs_optsl_optsrrrrroptsroptvalsimfilenamers r2Runrs |x| FF,IONT FF33JD$ % %S ( ( (CII $ $ $ GGGG / / /"OO + + + $   ( ( (!NN + + + $   a ?$5~GX  C00 (W - - - 0   h / / / 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0GGIIIHQKKKKKsC33C7 :C7 __main__)rrrr;r<r)rrr<r=) rrrr$rrrr$rr;r<r;r?)rLrMrrVrrVrrVrrVrrEr<rF)rrrr rrr<r=r)r)rr;r<r)rrr<r)TrB __future__rrr$ functoolsr"r rerr collectionsrcollections.abcrrrrr ior r r r typingrrrrrrrrrrrZrpylint.checkersrrrpylint.reporters.ureports.nodesrr pylint.typingrr pylint.utilsr r! pylint.lintr"rDcompilerr;r$r%r' HashToIndex_TIndexToLines_Trr6rr3r4rDLinesChunkLimits_Trrrrrrtotal_orderingrrr/rrrrrr,r0r1r2rsf 2#"""""  ######CCCCCCCCCCCC6666666666                        SSSSSSSSSS::::::::9999999955555555%$$$$$$)rz(33 W\3 ' ' *\4;./ e445V^W45 ========(02JJK$ $ $ $ $ $ $ $ NFFFFFFFF8        :9j*<=/J&#&#&#&#&#R((((VYYYYD*fVfVfVfVfVfVfVfV^@D ]]]]]@ 6 6 6 6 6 6 6 6 t +GGGGDDDDD'DDDN4444*****Z zCEEEEEr1