Insight Engine - AI Content Pipeline on AWS
Insight Engine is an AI-powered event-driven content pipeline developed for the AI for Bharat Hackathon 2025. The project was designed to automate the workflow from discovering research content to generating platform-optimized social media posts using scalable AWS cloud infrastructure and Generative AI. The primary objective of the project was to explore how modern AI systems, serverless architectures, and event-driven workflows can be combined to build scalable automation platforms capable of processing, analyzing, and generating content efficiently.
Built an event-driven AI content automation pipeline using AWS Bedrock, Lambda, DynamoDB, Terraform, and Next.js. The system automates the workflow from discovering research papers and RSS content to generating platform-optimized social media posts with human approval workflows.
During the AI for Bharat Hackathon 2025, our team wanted to explore how Generative AI and serverless cloud infrastructure could be combined to automate content workflows at scale. Instead of building a simple text-generation application, we focused on designing a complete event-driven pipeline capable of discovering, processing, analyzing, and generating content automatically.
The project we built was called Insight Engine — an AI-powered content pipeline designed to automate the journey from research discovery to social media content generation.
The system ingests content from RSS feeds and arXiv research papers, processes the incoming data, filters duplicate content using semantic similarity, and generates platform-specific posts for LinkedIn and Twitter/X using Generative AI models on AWS Bedrock.
The backend architecture was designed using a serverless event-driven approach on AWS. AWS Lambda functions handle asynchronous processing workflows, while Amazon SQS and SNS manage communication between services. DynamoDB is used for storing workflow metadata, generated content, and processing states.
To improve content quality and reduce duplicate outputs, Titan Embeddings were used for semantic deduplication and similarity checks across incoming content. AWS Bedrock was integrated for relevance scoring, trend analysis, and AI-powered content generation.
One of the major goals of the project was maintaining human oversight while automating content generation. To solve this, we built a dashboard using Next.js 14 where generated posts could be reviewed, edited, and approved before publishing.
Authentication and access management were implemented using AWS Cognito, while Terraform was used to provision and manage the complete AWS infrastructure through Infrastructure as Code practices.
Deployment workflows were automated using GitHub Actions, which handled testing and deployment pipelines for backend services and infrastructure updates. The frontend dashboard was deployed using Vercel.
This project gave me practical exposure to building scalable event-driven architectures, integrating Generative AI into backend systems, designing serverless workflows, and managing infrastructure automation on AWS.
Insight Engine was developed for the AI for Bharat Hackathon 2025, where our team was shortlisted among the Top 36 teams in the Student Track from over 16,000–30,000 participating developers.
Technologies Used
Node.js • TypeScript • AWS Lambda • DynamoDB • SQS • SNS • AWS Bedrock • Terraform • Next.js • AWS Cognito • GitHub Actions • Vercel • Event-Driven Architecture • Generative AI