Scrape webelements

Question

I have been working with requests where I as easy as it is. Scraping the webpage and add it into a dict and print the payload if we find a new value or not.

import json
import re

import requests
from selectolax.parser import HTMLParser

payload = {
    "store": "Not found",
    "name": "Not found",
    "price": "Not found",
    "image": "Not found",
    "sizes": []
}

response = requests.get("https://shelta.se/sneakers/nike-air-zoom-type-whiteblack-cj2033-103")

if response.ok:

    bs4 = HTMLParser(response.text)

    product_name = bs4.css_first('h1[class="product-page-header"]')
    product_price = bs4.css_first('span[class="price"]')
    product_image = bs4.css_first('meta[property="og:image"]')

    if product_name:
        payload['name'] = product_name.text().strip()

    if product_price:
        payload['price'] = "{} Price".format(product_price.text().strip())

    if product_image:
        payload['image'] = product_image.attrs['content']

    try:
        attribute = json.loads(
            '{}'.format(
                re.search(
                    r'var\s*JetshopData\s*=\s*(.*?);',
                    response.text,
                    re.M | re.S
                ).group(1)
            )
        )

        payload['sizes'] = [
            f'{get_value["Variation"][0]}'
            for get_value in attribute['ProductInfo']['Attributes']['Variations']
            if get_value.get('IsBuyable')
        ]

    except Exception:  # noqa
        pass

    del bs4
    print("New payload!", payload)

else:
    print("No new payload!", payload)

The idea mostly is that if we find the values then we want to replace the values in the dict and if we dont find it, basically skip it.

Things that made me concerned:

1. What happens if one of the if statements fails? Fails I mean etc if I cannot scrape product_image.attrs['content'] - That would end up in a exception where it stops the script which I do not want to do.
2. Im pretty sure to use except Exception: # noqa is a bad practice and I do not know what would be the best practice to handle it.

I would appreciate all kind of tips and tricks and how I can improve my knowledge with python!

Answer 1

• You have not described the purpose of payload. If this is a JSON payload going to some other web service, Not found is a poor choice for a missing value and None would be more appropriate
• You never use payload['store']
• Your selector h1[class="product-page-header"] - which is xpath (?) syntax - can just be h1.product-page-header in CSS selector syntax
• I think your regex for JetshopData is unnecessarily permissive. If the format breaks, you should be notified by a parse failure rather than silently letting a changed design through - since the outer dictionary format will likely not be the only thing to change
• You should constrain your regex to only looking in <script> values rather than through the entire HTML document
• '{}'.format is redundant
• Tell requests when you're done with the response via context management; conversely, there is no benefit to del bs4 if you have proper method scope
• It's likely that you should be looking at all variations instead of just the first
• Don't blanket-except. If you get a specific exception that you want to ignore that, ignore that in a narrow manner.
• Separate your scraping code from your payload formation code

The following suggested code uses BeautifulSoup because it's what I'm more familiar with and I didn't want to bother installing selectolax:

import json
import re
from dataclasses import dataclass
from pprint import pprint
from typing import Optional, List

import requests
from bs4 import BeautifulSoup


@dataclass
class Product:
    name: Optional[str]
    price: Optional[str]
    image: Optional[str]
    sizes: List[str]

    @staticmethod
    def get_sizes(doc: BeautifulSoup) -> List[str]:
        pat = re.compile(
            r'^<script>var JetshopData='
            r'(\{.*\})'
            r';</script>$',
        )
        for script in doc.find_all('script'):
            match = pat.match(str(script))
            if match is not None:
                break
        else:
            return []

        data = json.loads(match[1])
        return [
            variation
            for get_value in data['ProductInfo']['Attributes']['Variations']
            if get_value.get('IsBuyable')
            for variation in get_value['Variation']
        ]

    @classmethod
    def from_page(cls, url: str) -> Optional['Product']:
        with requests.get(url) as response:
            if not response.ok:
                return None
            doc = BeautifulSoup(response.text, 'html.parser')

        name = doc.select_one('h1.product-page-header')
        price = doc.select_one('span.price')
        image = doc.select_one('meta[property="og:image"]')

        return cls(
            name=name and name.text.strip(),
            price=price and price.text.strip(),
            image=image and image['content'],
            sizes=cls.get_sizes(doc),
        )

    @property
    def payload(self) -> dict:
        return {
            "name": self.name or "Not found",
            "price": self.price or "Not found",
            "image": self.image or "Not found",
            "sizes": self.sizes,
        }


def main():
    product = Product.from_page("https://shelta.se/sneakers/nike-air-zoom-type-whiteblack-cj2033-103")
    if product is None:
        print('No new payload')
    else:
        print('New payload:')
        pprint(product.payload)


if __name__ == '__main__':
    main()

Answer 2

Naming confusion

First of all there is something that is confusing in your code: bs4 does not actually represent an instance of BeautifulSoup. Pretty much any Python code that is based on BeautifulSoup has a line like this:

from bs4 import BeautifulSoup

But in your code bs4 represents something else: HTMLParser. So I would use a different name, to make it clear that it indeed is not BS, and thus the methods available are completely different. I was confused with this:

bs4.css_first

because I am not familiar with such a method being present in BS4, and for a reason. The documentation shows that it behaves like find in BS and returns None if not matching element is found.

You've raised one obvious issue, what happens if one (or more) of the value you are attempting to retrieve is not there ? Obviously you should test the results because sooner or later the website design will change and your code will stop working as expected. Simply test that all your values are different than None using any or more straightforward:

if None in (product_name, product_price, product_image):

If this condition is met, then your code should stop. In this code you are retrieving only 3 elements but you could have made a loop (handy if your list is going to be longer). All you need is a basic dict that contains your xpath queries, like:

product_search = {
    "name": 'h1[class="product-page-header"]',
    "price": 'span[class="price"]',
    "image": 'meta[property="og:image"]'
}

And if any of the xpath queries does not succeed, then you break out of the loop immediately.

Exception handling

Something that one should never do:

except Exception:  # noqa
    pass

If an exception occurs, at least log it or print it. Do not discard it, do not hide it. This practice can only lead to unreliable applications, that are hard to debug. If you can, avoid the exception. For instance you can easily anticipate that your xpath queries may fail, but it's easy to check the results.

There is one type of exception you could handle, it is requests.exceptions. Because it's perfectly possible that the website could be down sometimes, or that your network connection is down, or the DNS is failing.

In general, it is good to have one generic exception handler for the whole application, and in certain sections of the code, you can add limited handling for specific types of exceptions. Thus, requests.get should be in a try block like:

try:
    response = requests.get("https://shelta.se/sneakers/nike-air-zoom-type-whiteblack-cj2033-103")
except requests.exceptions.RequestException as e:
    # print error message and stop execution

Note that in this example I am catching the base exception, but you could handle the different types of requests exceptions separately (eg: requests.exceptions.Timeout).