WorldPulse - Global News & Crisis Monitoring

Overview

WorldPulse is a web application featuring an interactive 3D globe for monitoring global crises, disasters, natural hazards, and ongoing events in real-time. The application aggregates data from multiple authoritative sources (GDACS, USGS, ThinkHazard!, ACLED, UCDP, CrisisWatch, ReliefWeb) and displays them on a Mapbox-powered globe interface with enterprise-grade visualization capabilities.

The application provides users with the ability to explore the globe, search for locations, filter events by type and severity, and view detailed information about each disaster or crisis event. WorldPulse also features:

Real-time news aggregation from 40+ global news sources via RSS/JSON feeds
AI-powered location research reports using Valyu DeepResearch
Subscription-based premium features with flexible pricing plans
Conflict event tracking from ACLED and UCDP databases
Image extraction for news articles

Project Screenshot

System Architecture

Frontend Architecture

Technology Stack:

React with TypeScript for component-based UI
Wouter for client-side routing
TanStack Query (React Query) for data fetching and caching
Mapbox GL JS with globe projection for 3D visualization
Shadcn UI component library built on Radix UI primitives
Tailwind CSS for styling with custom design system
Supabase Auth for authentication (Google, GitHub, email/password)

Key UI Components:

Full-screen globe as the primary interface
Fixed header with search and theme controls
Collapsible left sidebar for map controls and filters
Sliding right panel for event details
Event markers with severity-based color coding

State Management:

React Query for server state and caching (60-second refetch interval)
Local React state for UI controls (sidebar visibility, selected events, filters)
Supabase Auth state for user authentication

Backend Architecture

Technology Stack:

Node.js with Express.js for REST API server
TypeScript throughout the codebase
Drizzle ORM for type-safe database operations
Supabase for authentication and database
Development mode uses Vite middleware for HMR
Production mode serves pre-built static assets

API Design:

RESTful endpoints following resource-based patterns
/api/events - Get all active events
/api/events/:id - Get specific event details
/api/events/type/:type - Filter events by type
/api/config - Application configuration (Mapbox token)
/api/auth/me - Get current authenticated user
/api/news/feeds - Get all news feeds, create custom feeds
/api/news - Get news articles with feed information
/api/news/search - Search news with Valyu AI integration
/api/subscription/checkout - Create subscription checkout session
/api/subscription/webhook - Handle Polar webhooks
/api/research/location - Start location research, get reports

Authentication:

Supabase Auth with JWT-based stateless authentication
Supports Google OAuth, GitHub OAuth, and email/password
Backend validates JWT tokens from Authorization header

Scheduled Tasks: The application uses node-cron for scheduled data fetching and maintenance:

Event Data Fetching (every 6 minutes):
- GDACS natural disaster feeds
- USGS earthquake data
- UCDP conflict events
- ACLED conflict events
- CrisisWatch RSS feeds
- ReliefWeb humanitarian updates
- Automatic cleanup of events older than 7 days
News Feed Fetching (every 5 minutes):
- Fetches from all enabled RSS/JSON feeds
- Processes 40+ default feeds plus custom user feeds
- Image extraction and location geocoding
- Event deduplication and storage
Research Report Cleanup (every hour):
- Removes expired research reports (older than 5 days)
- Frees up database storage
Automatic Initialization:
- News feeds are automatically initialized on first run
- Default feeds are created if none exist in database

Data Processing:

Scheduled data fetching using node-cron:
- Event fetching every 6 minutes (GDACS, USGS, UCDP, ACLED, CrisisWatch, ReliefWeb)
- News feed fetching every 5 minutes
- Research report cleanup every hour
XML/RSS parsing for GDACS feeds and news sources
JSON parsing for USGS earthquake data and JSON Feed format
News feed aggregation with automatic initialization of 40+ default feeds
Image extraction from articles with multiple fallback strategies (RSS images, Open Graph, Twitter cards, JSON-LD, HTML parsing)
Location extraction from news text using geocoding
Event deduplication using SHA256 hashing of normalized URLs
Feed-specific event filtering and enable/disable functionality
Automatic cleanup of old inactive events and expired research reports

Data Storage

Database:

PostgreSQL via Supabase
Drizzle ORM for type-safe database operations

Schema Design:

Events Table:

Stores disaster events from multiple sources (GDACS, USGS, ThinkHazard!)
Geospatial data (latitude/longitude) for map plotting
Severity levels (green/yellow/orange/red) for visual prioritization
Temporal data (start date, end date, last update)
Metadata (affected population, magnitude, region, country)
Indexed on source, event type, severity, and active status for query performance

Hazard Types Table:

Reference data for ThinkHazard! categories
Stores display metadata (icons, colors) for UI rendering

Users Table:

id: Primary key (UUID)
supabaseId: Supabase Auth user ID
email: Normalized to lowercase for consistent lookups
username: Display name
password: Hashed password (for legacy email/password users)
firstName, lastName, profileImageUrl: Profile data from OAuth
subscriptionTier: 'free', 'premium', or 'pay_per_use'
subscriptionStatus: Active subscription status
polarCustomerId: Polar customer ID for subscription management
hasUsedTrial: Whether user has used the 3-day free trial
hazardLookupCount, newsLookupCount: Usage tracking for premium features

News Feeds Table:

Stores RSS/JSON feed definitions for news aggregation
Fields: name, url, type (rss/json), country, category, isDefault, enabled, createdBy
Supports custom user-created feeds
Automatic initialization of 40+ default feeds from major news sources

Location Reports Table:

Stores AI-powered research reports generated via Valyu DeepResearch
Fields: deepresearchId, locationName, locationLat, locationLng, countryCode, customQuery
Report content stored as markdown with sources, wordCount, sourceCount
Reports expire after 5 days (automatic cleanup)
Status tracking: queued, running, completed, failed

Fetch Logs Table:

Tracks data source reliability and fetch history
Records success/failure status and error messages

Environment Variables

Required Variables

# Supabase Configuration
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_ANON_KEY=your-anon-key
SUPABASE_SERVICE_ROLE_KEY=your-service-role-key

# Database (Supabase Postgres connection string)
DATABASE_URL=postgresql://postgres:password@db.your-project.supabase.co:5432/postgres

# Mapbox
MAPBOX_ACCESS_TOKEN=your-mapbox-token

# Polar Subscription Management
POLAR_ACCESS_TOKEN=your-polar-access-token
POLAR_WEBHOOK_SECRET=your-polar-webhook-secret
POLAR_SUBSCRIPTION_PRODUCT_ID=prod_xxxxxxxxxxxxxxxxxxxx
POLAR_PAY_PER_USE_PRODUCT_ID=prod_xxxxxxxxxxxxxxxxxxxx
APP_URL=https://your-domain.com  # For checkout redirects

# Conflict Data Sources
ACLED_USERNAME=your-acled-email
ACLED_PASSWORD=your-acled-password
# Optional: ACLED_ACCESS_TOKEN=your-manual-token  # If not using OAuth credentials

# News APIs
VALYU_API_KEY=your-valyu-api-key  # For news search and location research
NEWSDATA_API_KEY=your-newsdata-api-key  # Optional: NewsData.io API
MEDIASTACK_API_KEY=your-mediastack-api-key  # Optional: Mediastack API

# Optional
GROQ_API_KEY=your-groq-api-key  # For news summary generation
SESSION_SECRET=your-session-secret  # For legacy session support
PORT=5000  # Server port (default: 5000)
NODE_ENV=development  # or 'production'

Frontend Environment Variables (Vite)

Create a .env file in the project root with these variables prefixed with VITE_:

VITE_SUPABASE_URL=https://your-project.supabase.co
VITE_SUPABASE_ANON_KEY=your-anon-key

Getting Started

Prerequisites

Node.js 18+
A Supabase project with:
- Google OAuth provider enabled
- GitHub OAuth provider enabled
- Email/password auth enabled
A Mapbox account with API token

Installation

Clone the repository
Install dependencies:
```
npm install
```
Set up environment variables (see above)
Push the database schema:
```
npm run db:push
```
Start the development server:
```
npm run dev
```

Building for Production

npm run build
npm start

Key Features

News Feed System

WorldPulse includes a comprehensive RSS/JSON news feed aggregation system that automatically collects and processes news articles from major global sources.

Feed Support:

RSS 2.0, Atom, and JSON Feed format support
40+ default feeds from major news sources (BBC, CNN, Reuters, NYT, Al Jazeera, DW, Guardian, Sky News, NPR, ABC News, CBS News, and more)
Custom feed management: users can add their own RSS/JSON feeds
Feed filtering: enable/disable specific feeds per user
Automatic feed initialization on first run

Image Extraction: The system uses multiple fallback strategies to extract images from news articles:

RSS-provided images: Media RSS (media:content, media:thumbnail) and enclosure tags
Open Graph meta tags: Standard og:image property
Twitter card images: twitter:image meta tags
JSON-LD structured data: Image URLs from structured data
HTML parsing: First article image from HTML content (with semantic element prio

WorldPulse

Install / Use

README