itemadapter 0.0.1

Common interface for Scrapy items

itemadapter

The ItemAdapter class is a wrapper for data container objects, providing a common interface to handle objects of different types in an uniform manner, regardless of their underlying implementation.

This package started as an initiative to support dataclass objects as items in Scrapy. It was extracted out to a standalone package in order to allow it to be used independently.

Currently supported types are:

• Classes that implement the MutableMapping interface, including but not limited to:
  • scrapy.item.Item
  • dict
• dataclass-based classes
• attrs-based classes

Requirements

• Python 3.5+
• dataclasses (stdlib in Python 3.7+, or its backport in Python 3.6): optional, needed to interact with dataclass-based items
• attrs: optional, needed to interact with attrs-based items

API

ItemAdapter class

class itemadapter.ItemAdapter(item: Any)

ItemAdapter implements the MutableMapping interface, providing a dict-like API to manipulate data for the object it wraps (which is modified in-place).

Two additional methods are available:

get_field_meta(field_name: str) -> MappingProxyType

Return a MappingProxyType object with metadata about the given field, or raise TypeError if the item class does not support field metadata.

The returned value is taken from the following sources, depending on the item type:

• dataclasses.field.metadata for dataclass-based items
• attr.Attribute.metadata for attrs-based items
• scrapy.item.Field for scrapy.item.Items

field_names() -> List[str]

Return a list with the names of all the defined fields for the item.

is_item function

itemadapter.is_item(obj: Any) -> bool

Return True if the given object belongs to one of the supported types, False otherwise.

Metadata support

scrapy.item.Item, dataclass and attrs objects allow the inclusion of arbitrary field metadata, which can be retrieved with the ItemAdapter.get_field_meta method. The definition procedure depends on the underlying type.

scrapy.item.Item objects

>>> from scrapy.item import Item, Field
>>> from itemadapter import ItemAdapter
>>> class InventoryItem(Item):
...     name = Field(serializer=str)
...     value = Field(serializer=int, limit=100)
...
>>> adapter = ItemAdapter(InventoryItem(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'serializer': <class 'str'>})
>>> adapter.get_field_meta("value")
mappingproxy({'serializer': <class 'int'>, 'limit': 100})

dataclass objects

>>> from dataclasses import dataclass, field
>>> @dataclass
... class InventoryItem:
...     name: str = field(metadata={"serializer": str})
...     value: int = field(metadata={"serializer": int, "limit": 100})
...
>>> adapter = ItemAdapter(InventoryItem(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'serializer': <class 'str'>})
>>> adapter.get_field_meta("value")
mappingproxy({'serializer': <class 'int'>, 'limit': 100})

attrs objects

>>> import attr
>>> @attr.s
... class InventoryItem:
...     name = attr.ib(metadata={"serializer": str})
...     value = attr.ib(metadata={"serializer": int})
...
>>> adapter = ItemAdapter(InventoryItem(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'serializer': <class 'str'>})
>>> adapter.get_field_meta("value")
mappingproxy({'serializer': <class 'int'>})

Other types

In fact, any supported object with a fields attribute which values are mappings works:

>>> class DictWithFields(dict):
...     fields = {
...         "name": {"serializer": str},
...         "value": {"serializer": int, "limit": 100},
...     }
...
>>> adapter = ItemAdapter(DictWithFields(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'serializer': <class 'str'>})
>>> adapter.get_field_meta("value")
mappingproxy({'serializer': <class 'int'>, 'limit': 100})

Examples

scrapy.item.Item objects

>>> from scrapy.item import Item, Field
>>> from itemadapter import ItemAdapter
>>> class InventoryItem(Item):
...     name = Field()
...     price = Field()
...
>>> item = InventoryItem(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
{'name': 'bar', 'price': 5}

dict

>>> from itemadapter import ItemAdapter
>>> item = dict(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
{'name': 'bar', 'price': 5}

dataclass objects

>>> from dataclasses import dataclass
>>> from itemadapter import ItemAdapter
>>> @dataclass
... class InventoryItem:
...     name: str
...     price: int
...
>>> item = InventoryItem(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
InventoryItem(name='bar', price=5)

attrs objects

>>> import attr
>>> from itemadapter import ItemAdapter
>>> @attr.s
... class InventoryItem:
...     name = attr.ib()
...     price = attr.ib()
...
>>> item = InventoryItem(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
InventoryItem(name='bar', price=5)