dlt.extract.items_transform
ItemTransform Objects
class ItemTransform(BaseItemTransform[TCustomMetrics], ABC,
Generic[TAny, TCustomMetrics])
placement_affinity
Tell how strongly an item sticks to start (-1) or end (+1) of pipe.
__call__
@abstractmethod
def __call__(item: TDataItems, meta: Any = None) -> Optional[TDataItems]
Transforms item (a list of TDataItem or a single TDataItem) and returns or yields TDataItems. Returns None to consume item (filter out)
ValidateItem Objects
class ValidateItem(ItemTransform[TDataItem])
Base class for validators of data items.
Subclass should implement the __call__ method to either return the data item(s) or raise extract.exceptions.ValidationError.
See PydanticValidator for possible implementation.
placement_affinity
stick to end but less than incremental
LimitItem Objects
class LimitItem(ItemTransform[TDataItem])
placement_affinity
stick to end right behind incremental
limit
def limit(chunk_size: int) -> Optional[int]
Calculate the maximum number of rows to which result is limited. Limit works in chunks
that controlled by the data source and this must be provided in chunk_size.
chunk_size will be ignore if counting rows (count_rows is True). Mind that
this count method will not split batches so you may get more items (up to the full last batch)
than limit method indicates.
MetricsItem Objects
class MetricsItem(ItemTransform[None])
Collects custom metrics from data flowing through the pipe without modifying items.
The metrics function receives items, optionally meta, and a metrics dictionary that it can update in-place. The items are passed through unchanged.