Scrapy Item Ingest
  • Installation
    • Requirements
    • Install from PyPI
    • Minimal configuration (settings.py)
    • Run
    • Troubleshooting
    • Next steps
  • Quick Start
    • 1) Install
    • 2) Enable (settings.py)
    • 3) Run
    • 4) Verify
    • 5) Troubleshooting
  • Configuration
    • Required
    • Recommended
    • Optional
    • Table names (optional)
    • Logging to DB (optional)
    • Tips
  • Basic Recipe: Items + Requests + Logs
    • 1) Install
    • 2) Enable (settings.py)
    • 3) Run
    • Expected tables
    • Tips
  • Recipe: Items-only pipeline
    • 1) Enable (settings.py)
    • 2) Run
    • Expected
    • Tip
  • Recipe: Requests tracking
    • 1) Enable (settings.py)
    • 2) What it logs
    • 3) Run
    • Expected
    • Tip
  • Recipe: Scrapy logs to DB
    • 1) Enable (settings.py)
    • 2) Run
    • Expected
    • Tips
  • Troubleshooting
    • Common Installation Issues
      • PostgreSQL Connection Errors
      • Table Creation Issues
    • Runtime Issues
      • Memory-Related Problems
      • Performance Issues
    • Configuration Problems
      • Pipeline Order Issues
      • Job ID Configuration Issues
    • Data Quality Issues
      • JSON Serialization Errors
      • JSONB Query Issues
    • Monitoring and Debugging
      • Enable Debug Logging
      • Database Connection Debugging
      • Performance Profiling
    • Common Error Messages and Solutions
      • ImportError: No module named 'scrapy_item_ingest'
      • AttributeError: 'Spider' object has no attribute 'job_id'
      • psycopg2.errors.UndefinedTable: relation "job_items" does not exist
    • Getting Help
    • Next Steps
  • Changelog
    • [Unreleased]
    • [0.2.4] - 2025-11-24
    • [0.2.3] - 2025-11-21
    • [0.2.2] - 2025-11-21
    • [0.2.0] - 2025-11-11
    • [0.1.1] - 2025-07-21
Scrapy Item Ingest
  • Overview: module code

All modules for which code is available

  • scrapy_item_ingest.config.settings
  • scrapy_item_ingest.extensions.base
  • scrapy_item_ingest.extensions.logging
  • scrapy_item_ingest.pipelines.base
  • scrapy_item_ingest.pipelines.items
  • scrapy_item_ingest.pipelines.main
  • scrapy_item_ingest.pipelines.requests
  • scrapy_item_ingest.utils.fingerprint
  • scrapy_item_ingest.utils.serialization
  • scrapy_item_ingest.utils.time

© Copyright 2025, Fawad Ali.

Built with Sphinx using a theme provided by Read the Docs.