scrapxd: The Library for Letterboxd Data

scrapxd is a Python library designed for web scraping, analyzing, and exporting data from Letterboxd, the social network for cinephiles. With an intuitive, strictly-typed API using Pydantic, scrapxd makes it easy to access user profiles, film lists, diaries, and much more.

Key Features

• Scrape Whatever You Need: Extract detailed data from user profiles, including watched films, ratings, diary entries, lists, followers, and more.
• Film Search: Search for films on Letterboxd based on various filters.
• Pydantic Data Models: All returned data is validated and structured into Pydantic models, ensuring consistency and ease of use in your code.
• Analytics Module: Perform statistical analysis on the collected data, such as correlations and trends (requires the [analytics] extra).
• Exporting Module: Export collected data to popular formats like CSV, JSON, and Excel (.xlsx) (requires the [export] extra).
• Retries Logic: Utilizes tenacity for automatic retries on network failures, making the scraping process more reliable.
• Simple and Intuitive: Designed with a clean and easy-to-use API, as demonstrated in the examples.

Installation

You can install the library directly from PyPI.

Standard Installation:

Bash

pip install scrapxd

The library has optional dependencies for extra features. You can install them as needed:

For Data Analytics:

Bash

pip install "scrapxd[analytics]"

For File Exporting:

Bash

pip install "scrapxd[export]"

To Install Everything (including testing dependencies):

Bash

pip install "scrapxd[all]"

Quickstart

Using scrapxd is very simple. Here is a basic example to get a user's watched films:

Python

from scrapxd import Scrapxd

# 1. Create a client instance
client = Scrapxd()

# 2. Get data for a Letterboxd user
# The client handles searching and pagination automatically
user = client.get_user("your_username_here")
user_films = user.logs

# 3. Access the data
print(f"Total films watched by '{user_films.username}': {user_films.number_of_entries}")

# Each entry is a Pydantic object with structured data
for entry in user_films.entries[:5]: # Displaying the first 5
    print(f"- {entry.film.title} ({entry.film.year}) - Rating: {entry.rating}")

# 4. (Optional) Export the data to an Excel file
try:
    user_films.to_xlsx(f"{user_films.username}_films")
    print(f"\nData exported to {user_films.username}_films.xlsx")
except ImportError:
    print("\nTo export data, please install with: pip install \"scrapxd[export]\"")

Detailed Examples

For a more in-depth guide covering all features like profile analysis, comparisons, and advanced use cases, please explore the Jupyter notebooks in the /examples folder:

• 1. Quickstart Guide
• 2. Deep Dive Analysis
• 3. Comparing Profiles
• 4. Advanced Guide

Contributing

Contributions are very welcome! If you have an idea for a new feature, find a bug, or want to improve the documentation, please open an Issue or submit a Pull Request.

License

This project is licensed under the MIT License. See the LICENSE file for more details.

Contact

Cauã Santos - My LinkedIn Profile - cauafsantosdev@gmail.com

GitHub URL: https://github.com/cauafsantosdev/scrapxd