o Dh@sdZddlZddlmZmZddlmZmZmZddl m Z ddl m Z ddl mZmZdd lmZd e d e fd d Zded eeeeffddZddeded efddZdS)z< Module regrouping baseline and basic extraction functions. N)AnyTuple)_ElementElement SubElement) HtmlElement)BASIC_CLEAN_XPATH) load_htmltrim)delete_elementtreereturncCst|D]}t|q|S)z-Remove a few section types from the document.)r r )r elemrU/home/air/segue/gemini/back/venv/lib/python3.10/site-packages/trafilatura/baseline.pybasic_cleanings  r filecontentc CsVt|}td}|dur|ddfSd}|dD]M}|jred|jvrez t|jdd}Wn ty9d}Ynw|red|vrQt|}|durNt| nd}nt|}|t |d_||rbd |n|7}qt |d krs||t |fSt |}d}|d D]}t| }t |d kr|t |d_||rd |n|7}q~t |dkr||t |fSt } d}|d d ddddD]!} t| } | | vr| t |d_||rd | n| 7}| | qt |d kr||t |fStd}|d} | durt |d} dd| D}ddd|D| _|| jt | jfSt|dd}|t |d_||t |fS)a)Use baseline extraction function targeting text paragraphs and/or JSON metadata. Args: filecontent: HTML code as binary string or string. Returns: A LXML element containing the extracted paragraphs, the main text as string, and its length as integer. bodyNrz&.//script[@type="application/ld+json"] articleBodyz

p dz .//article blockquotecodepreqquote.//bodycSsg|]}t|qSr)r .0errr ^zbaseline.. cSsg|]}|r|qSrrr rrrr#_r$F)clean)r riterfindtextjsonloadsget Exceptionr text_contentrlenrsetiteraddfinditertextjoinhtml2txt)rr postbody temp_textr json_bodyparsedr( article_elemresultselemententry body_elemp_elem text_elemsrrrbaselinesj                  rATcontentr&cCsLt|}|dur dS|d}|durdS|rt|}d|S)zRun basic html2txt on a document. Args: content: HTML document as string or LXML element. clean: remove potentially undesirable elements. Returns: The extracted text in the form of a string or an empty string. Nrrr)r r2rr4r-splitstrip)rBr&r rrrrr5hs  r5)T)__doc__r)typingrr lxml.etreerrr lxml.htmlrsettingsr utilsr r xmlr rstrintrAboolr5rrrrs   O