Various utility functions and classes for working with the HTML soup from beautifulsoup4.

Short summary:

• as_xml: Transform tag into an lxml XML element.

• child_tags: A generator yielding the immediate child tags of child whose tag name is child_name. If child_name is None, yield all the immediate child tags, skipping things like strings and comments.

• find_heading: Find the nearest heading satisfying the test filter(tag). Return the tag or None is one is not found. The default filter tests that the heading is not empty. This uses find_up to locate the tag.

• find_up: A generator yielding (found,ref) 2-tuples obtained by search the tag tree left and up from tag using .previous_sibling and .parent, matching tags where test(found) is true.

• printt_soup: Print the contents of the soup via cs.lex.printt using tabulate_soup to make the table.

• Table: A Widget subclass representing an HTML TABLE tag.

• tabulate_soup: Return a table describing soup for use with cs.lex.printt. Connect tags with their child tags using Unicode box characters.

• Widget: Base class for various "widget" HTML constructs, such as a TABLE, or in principle anything else regular on a page.

Functions

as_xml(tag: bs4.element.Tag, *, E=None)

Transform tag into an lxml XML element.

child_tags(tag, child_name: str | None = None) -> Iterable[bs4.element.Tag]

A generator yielding the immediate child tags of child whose tag name is child_name. If child_name is None, yield all the immediate child tags, skipping things like strings and comments.

find_heading(tag, filter: Callable[[bs4.element.Tag], bool] = <function

Find the nearest heading satisfying the test filter(tag). Return the tag or None is one is not found. The default filter tests that the heading is not empty. This uses find_up to locate the tag.

find_up(tag, test: Union[str, Callable[[bs4.element.Tag], bool]], *, first=False) -> Generator[tuple[bs4.element.Tag, bs4.element.Tag] | tuple[None, None], NoneType, NoneType]

A generator yielding (found,ref) 2-tuples obtained by search the tag tree left and up from tag using .previous_sibling and .parent, matching tags where test(found) is true.

The test may be a tag .name value (a string) or a callable to evaluate a ound tag.

If first is true (default False) then the search stops after the first match. If there are no matches the tuple (None,None) is returned (this does not happen if first is false).

A primary use case for this is to find the heading tag for tag.

In the tuple, found is the matched tag. ref is the reference tag, the later sibling of found where the search started for that level; if found is at the same level as tag then ref will be tag.

For example, to locate the level 2 heading governing a tag:

(h2,_), *_ = find_up(tag,lambda found: found.name == 'h2')

or more concisely:

(h2,_), *_ = find_up(tag, 'h2')

or even:

(h2,_), = find_up(tag, 'h2',first=True)

Note that the first two will cause Python to raise an exception if there are no matches, while the third will provide h2 as None.

printt_soup(tag: bs4.element.Tag, *, with_ancestors=False, without_children=False, **printt_kw)

Print the contents of the soup via cs.lex.printt using tabulate_soup to make the table.

tabulate_soup(tag: bs4.element.Tag | bs4.element.NavigableString, *, with_ancestors=False, without_children=False) -> list[list[str, str] | tuple]

Return a table describing soup for use with cs.lex.printt. Connect tags with their child tags using Unicode box characters.

Parameters:

• tag: the Tag or NovigableString to tabulate
• with_ancestors: if true (default False), prepend the ancestor tags of tag
• without_children: if true (default False), omit the children of a Tag

Classes

class Table(Widget)

A Widget subclass representing an HTML TABLE tag.

Table.__init__(self, tag)

Scan the TABLE for the basic structures, used for the other properties etc later.

Note that if there was no TBODY, the immediate rows of the TABLE are presented as though they were in a single TBODY.

Table.all_rows

<functools.cached_property object at 0x10d5cb8d0>

Table.as_indexed_values(self, *, convert: Optional[Callable[[str, int, int, int, bs4.element.Tag], Any]] = None, convert_head_cell: Optional[Callable[[str, int, int, int, bs4.element.Tag], Any]] = None, convert_body_cell: Optional[Callable[[str, int, int, int, bs4.element.Tag], Any]] = None, convert_foot_cell: Optional[Callable[[str, int, int, int, bs4.element.Tag], Any]] = None, omit_header=False, omit_footer=False) -> list[list[tuple[str, int, int, int, bs4.element.Tag, typing.Any]]]

Return the table contents as a list-of-lists of indexed cell values. Each inner list contains the cell value records from a row.

this is an elaborate counterpart to the as_lists method.

Parameters:

• convert: the default cell conversion function
• convert_head: the header cell conversion function, default from convert
• convert_body: the body cell conversion function, default from convert
• convert_foot: the footer cell conversion function, default from convert
• omit_header: do not include rows from the THEAD section
• omit_footer: do not include rows from the TFOOT section

The conversion functions accept the following positional parameters:

• section_type: one of "THEAD", "TBODY" or "TFOOT"
• section_index: the index of the section, 0 for the header or footer but there may be multiple TBODY sections
• row_index: the index of the row within the section
• col_index: the index of the column within the row
• cell: the TD or TH tag for the cell The function should return the converted value of cell. The default conversion function returns cell.

The row and column indices supplied to the conversion unction are of the resolved cells, after expansion via the colspan or rowspan values. For example, a row with 3 cells whose second cell had a colspan=2 would be a list of 4 cells, with the second original cell referenced in the second and third items of the list; it will be the same tag instance.

Each cell instance is converted only once; the same cell spanning multiple columns or rows will have the same value instance in the result record.

The resulting list-of-lists contains value records, a 6-tuple of (section_type,section_index,row_index,col_index,cell,value). Note that the row_index and col_index are those of the top left index where the cell was first encountered for cells spanning multiple columns or rows.

Examples:

Convert every numeric cell to its float value, leave other cells as their text.

def as_float(section_type, section_index, row_index, column_index, cell):
    text = cell.get_text.strip()
    try:
        value = float(text)
    except ValueError:
        value = text
    return value

values = T.as_indexed_values(convert=as_float)

Convert only the body cells, keep the headers as tags, omit the footer:

values = T.as_indexed_values(convert_body_cell=as_float, omit_footer=True)

Table.as_lists(self, *, omit_header=False, omit_footer=False) -> list[list[bs4.element.Tag]]

Return the table contents as a list-of-lists-of-tags; each inner list is a row of tags. The innermost elements are the TH or TD tags. Note that cells spanning multiple columns or rows via their colspan or rowspan are the same reference.

Parameters:

• omit_header: do not include rows from the THEAD section
• omit_footer: do not include rows from the TFOOT section

Table.body_rows

<functools.cached_property object at 0x10d5cb3d0>

Table.cell_colspan(cell: bs4.element.Tag) -> int

Compute the colspan value for a table cell.

Table.cell_rowspan(cell: bs4.element.Tag) -> int

Compute the rowspan value for a table cell.

Table.foot_rows

<functools.cached_property object at 0x10d5cb650>

Table.head_rows

<functools.cached_property object at 0x10d5cb350>

Table.printt(self)

Print the table text.

Table.row_cells(tr: bs4.element.Tag) -> list[bs4.element.Tag]

Return a list of the cells (TD or TH) from a TR tag. colspan is supported by referencing the same cell multiple times. Only TD and TH tags which are immediate children of the TR are recognised.

Table.section_rows(section: bs4.element.Tag | None) -> list[list[bs4.element.Tag]]

Return the rows from a table section such as THEAD, TBODY, or TFOOT. rowspan is supported by referencing the same cell in lower rows.

Table.title

<functools.cached_property object at 0x10d5cb950>

class Widget

Base class for various "widget" HTML constructs, such as a TABLE, or in principle anything else regular on a page.

A Widgwt supplies:

• __init__(tag) to record the target BS4 tag, typically the top level tag encompassing the widget
• find_all(soup): returning a list of the top level tags within the BS4 tag soup; the default method calls soup.find_all() with the lower case version of the class name via soup.find_all()
• scan(soup): a factory method calling cls(tag) for every tag found by find_all(soup)

Everything else in a subclass supports whatever needs doing with the widget; the Table class is an exemplar:

• its __init__ method passes the tag to super().__init__() as normal, then finds a few top level things about the table
  • the caption, header, bodies, footer
• the default find_all is used because the lass name matches the HTML tag name
• everything else more complex is provided as methods or @cached_property properties, computed on demand

Widget.__init__(self, tag: bs4.element.Tag)

Initialise this Widget by saving tag as self.tag and then calling self.scan().

Widget.find_all(soup) -> list[bs4.element.Tag]

The default find_all finds tags from soup whose name matches the class name.

Widget.scan(soup) -> list[typing.Self]

Return a list of all Widgets of this type found in soup.