Soupy

Soupy is a wrapper around BeautifulSoup that makes it easier to build complex queries when wrangling web data.

Here's an example of a Soupy query.

from soupy import Soupy, Q

html = """
<div id="main">
  <div>The web is messy</div>
  and full of traps
  <div>but Soupy loves you</div>
</div>"""

print(Soupy(html).find(id='main').children
      .each(Q.text.strip()) # extract text from each node, trim whitespace
      .filter(len)          # remove empty strings
      .val())               # dump out of Soupy

# ['The web is messy', 'and full of traps', 'but Soupy loves you']

The same query using BeautifulSoup:

 from bs4 import BeautifulSoup, NavigableString

html = """
<div id="main">
  <div>The web is messy</div>
  and full of traps
  <div>but Soupy loves you</div>
</div>"""

result = []
for node in BeautifulSoup(html).find(id='main').children:
    if isinstance(node, NavigableString):
        text = node.strip()
    else:
        text = node.text.strip()
    if len(text):
        result.append(text)

print(result)

For more information, see the Soupy Documentation

Installation

pip install soupy

Dependencies

six and BeautifulSoup4

Soupy is supported on Python 2.6+ and 3.3+