Automated News Crawling Script

I need a robust script that continuously crawls online news sources, captures the full article content, and stores the data in a structured way so I can run downstream analyses on topics, sentiment, and trends. My priority is reliability at scale, so the crawler must be able to:

• Run on a schedule of my choosing without manual triggers
• Render and collect content from sites that rely on dynamic, JavaScript-loaded elements
• Finish each crawl quickly and efficiently to minimise server load and bandwidth

Python is my default environment, and I already work with Scrapy, Selenium, BeautifulSoup, and pandas, but I’m open to other modern frameworks if they achieve better throughput or easier maintenance. The final deliverable should include well-commented code, a brief setup guide, and a sample dataset proving the crawler works across at least three major news outlets.

If you have solid experience building similar high-performance crawlers—as opposed to general web-scraping projects—please outline that background in your bid.