jiatangzhi/gcp-data-pipeline-security-monitoring
GCP data pipeline for security monitoring — ingests events via CSV or Pub/Sub, validates and loads them into BigQuery (raw → cleaned → aggregated). Orchestrated with Airflow, deployed to Cloud Run via Cloud Build, provisioned with Terraform. Includes a Claude-powered agent for natural language queries with BigQuery ML risk scoring.
Platform-specific configuration:
{
"mcpServers": {
"gcp-data-pipeline-security-monitoring": {
"command": "npx",
"args": [
"-y",
"gcp-data-pipeline-security-monitoring"
]
}
}
}Add the config above to .claude/settings.json under the mcpServers key.
> Stack: Python · Pandas · BigQuery · Apache Airflow · Docker · Terraform · Claude AI
---
┌─────────────────────────────────────────────────────────────────────────┐
│ GCP Security Monitoring Pipeline │
└─────────────────────────────────────────────────────────────────────────┘
┌───────────┐ ┌────────────────┐ ┌─────────────────┐
│ CSV Files │ │ Simulated API │ │ Google Sheets │
│ events │ │ (httpbin.org) │ │ (Mock client) │
│ logins │ └───────┬────────┘ └────────┬────────┘
│ transact. │ │ │
└─────┬─────┘ │ │
│ │ │
└──────────────────▼──────────────────────┘
│
┌──────▼──────┐
│ INGESTION │ ingest.py
│ ingest_all │
└──────┬──────┘
│
┌──────▼──────────┐
│ TRANSFORMATION │ transform.py
│ • Normalize │
│ • Deduplicate │
│ • Compute DAU │
│ • Failed rate │
│ • Suspicious │
└──────┬──────────┘
│
┌──────▼───────┐
│ VALIDATION │ validate.py
│ • No nulls │
│ • No dups │
│ • Timestamps│
└──────┬───────┘
│
┌────────────▼──────────────────┐
│ GOOGLE BIGQUERY │
│ project.security_monitoring │
│ ┌──────────────────────────┐ │
│ │ raw_events │ │
│ │ raw_logins Loading reviews...