← Back to projects
VMS Ingestion Platform
Healthcare staffing data pipeline
Overview
Modular pipeline ingesting jobs, placements, and candidates from multiple VMS systems with retry, deduplication, and monitoring.
Key contributions
- Scrapy + Selenium dynamic crawling with retry and deduplication.
- Redis caching, S3 storage, Slack failure alerts.
- Auth tokens managed via AWS Secrets Manager.
Tech stack
Interested in similar work?