Jawaban singkat: otomasi riset kompetitor = n8n workflow: (1) Discovery: Seed keyword → SERP → Competitor list, (2) Content Audit: Competitor sitemap + RSS → Topic extraction, (3) Gap Analysis: AI semantic match vs your content → Gap scoring, (4) Opportunity Prioritization: Traffic × Intent × KD Inverse × Business Value, (5) Content Brief Generation: AI generate outline + keyword + structure, (6) Scheduling: Auto-create ke Content Pipeline. Mingguan otomatis. Artikel ini implementasi lengkap + prompt + scheduling.
Mengapa Otomasi Riset Kompetitor Butuh n8n + AI (Bukan Manual)
| Aspek | Manual (Ahrefs/SEMrush UI + Spreadsheet) | n8n + AI Otomatis |
|---|---|---|
| Frekuensi | Bulanan (kapok manual) | Mingguan / Harian (Cron Otomatis) |
| Cakupan Kompetitor | 5-10 (Manual pilih) | 20-50 (Auto-discovery dari SERP + Organic Competitor) |
| Content Gap Detection | Manual compare topic list | AI Semantic Match + Gap Scoring (Embedding Similarity) |
| Keyword Gap Prioritization | Filter KD, Volume manual | AI: Gap Score = Volume × Intent × KD Inverse × Business Value |
| Content Brief Generation | Manual buat outline per artikel | AI: Auto-generate Outline + Keyword + Structure + E-E-A-T Angle |
| Time to Action | Minggu (Riset → Brief → Assign) | Menit (Gap → Brief → Content Pipeline → Assign Writer) |
| Consistency | Bervariasi (tergantung mood/analis) | 100% Konsisten (Prompt + Scoring Tetap) |
Arsitektur Otomasi Riset Kompetitor (n8n Workflow “Competitor_Research_Automation”)
1. TRIGGER → Cron Weekly (Senin 05:00) + Manual Trigger (Telegram “/riset”)
↓
2. COMPETITOR DISCOVERY (Parallel)
→ Seed Keywords (dari Sheets “Seed_Keywords”) → SERP API (Top 10) → Ahrefs/DataForSEO Organic Competitors → Union → Top 20 Competitor
→ Filter: DA > 20, Traffic > 1k, Niche Relevansi > 0.7
→ Seed Keywords (dari Sheets “Seed_Keywords”) → SERP API (Top 10) → Ahrefs/DataForSEO Organic Competitors → Union → Top 20 Competitor
→ Filter: DA > 20, Traffic > 1k, Niche Relevansi > 0.7
↓
3. CONTENT INGESTION (Parallel per Competitor)
→ Sitemap + RSS Feed → Crawl Top Pages (Traffic > 100/bln) → Extract: Title, H1, H2, Content, Word Count, Publish Date, Keywords, Entities
→ Sitemap + RSS Feed → Crawl Top Pages (Traffic > 100/bln) → Extract: Title, H1, H2, Content, Word Count, Publish Date, Keywords, Entities
↓
4. YOUR CONTENT BASELINE
→ Fetch Your Content (Content_Pipeline Sheets) → Extract Same Fields → Build Your Topic/Entity Index
→ Fetch Your Content (Content_Pipeline Sheets) → Extract Same Fields → Build Your Topic/Entity Index
↓
5. GAP ANALYSIS ENGINE (4 Parallel AI Nodes)
→ Content Gap: Topic/Entity Semantic Match vs Your Index → Gap Score
→ Keyword Gap: Competitor Ranking Keywords vs Your Rankings → Gap Score
→ Format Gap: Competitor Content Format (Video, Tool, Calculator) vs You
→ Freshness Gap: Competitor Update Frequency vs Your Content Age
→ Content Gap: Topic/Entity Semantic Match vs Your Index → Gap Score
→ Keyword Gap: Competitor Ranking Keywords vs Your Rankings → Gap Score
→ Format Gap: Competitor Content Format (Video, Tool, Calculator) vs You
→ Freshness Gap: Competitor Update Frequency vs Your Content Age
↓
6. AI PRIORITIZATION & BRIEF GENERATION
→ Gap Scoring: Traffic × Intent × KD Inverse × Business Value
→ AI Brief Generator: Outline + Target Keyword + Structure + E-E-A-T Angle + Monetization Hook
→ Dedup & Consolidate Cross-Competitor Gaps
→ Gap Scoring: Traffic × Intent × KD Inverse × Business Value
→ AI Brief Generator: Outline + Target Keyword + Structure + E-E-A-T Angle + Monetization Hook
→ Dedup & Consolidate Cross-Competitor Gaps
↓
7. OUTPUT & INTEGRATION
→ Sheets “Competitor_Gap_Report” + “Keyword_Gap_Report” + “Content_Briefs”
→ Auto-create Rows di “Content_Pipeline” (Status=PLANNED, Assigned=AI/Writer)
→ Telegram Alert: “📊 Weekly Competitor Gap: 23 Gap, 12 High Priority. Top: [Topic 1], [Topic 2].”
→ Sheets “Competitor_Gap_Report” + “Keyword_Gap_Report” + “Content_Briefs”
→ Auto-create Rows di “Content_Pipeline” (Status=PLANNED, Assigned=AI/Writer)
→ Telegram Alert: “📊 Weekly Competitor Gap: 23 Gap, 12 High Priority. Top: [Topic 1], [Topic 2].”
Implementasi Detail: 4 Gap Analysis Engine
1. Content Gap Engine (Topic & Entity Level)
Input: Competitor Topics (dari Crawl) + Your Topics (Content_Pipeline)
Process:
- Topic Extraction: AI Topic Modeling (BERTopic / LDA / AI Clustering) per Competitor URL → Primary Topic, Sub-topics, Entities
- Semantic Matching: Embedding (text-embedding-3-small) → Cosine Similarity Your Topics vs Competitor Topics
- Gap Classification:
- MISSING: Competitor punya, Anda TIDAK (Similarity < 0.3)
- THIN: Competitor Punya + Anda Punya tapi Depth Score Kompetitor > Anda + 30
- OUTDATED: Competitor Updated < 30 hari, Anda > 90 hari
- FORMAT_GAP: Competitor Punya Video/Tool/Calculator, Anda Hanya Teks
- Gap Scoring: Score = (Competitor Traffic × 0.4) + (Total Keyword Volume × 0.3) + (Content Depth Gap × 0.2) + (Recency × 0.1)
2. Keyword Gap Engine (Ranking Opportunity)
Input: Competitor Organic Keywords (API) vs Your Keywords (GSC/API)
- Gap Identification: Keyword yang Competitor Rank Pos 1-20 TAPI Anda Not Rank / Rank > 50
- AI Enrichment per Keyword:
- Intent Classification (Informational/Commercial/Transactional/Navigational)
- KD Re-Analysis (SERP-based: DA, Content Depth, Backlink, UX)
- Content Format Recommended (Article/Video/Tool/Comparison/Calculator)
- Monetization Potential (Affiliate/Ads/Lead/Product)
- Gap Scoring: Score = (Competitor Position Weight × 0.3) + (Volume × 0.25) + (KD Inverse × 0.2) + (Intent Match Business × 0.15) + (SERP Feature Opportunity × 0.1)
3. Format Gap Engine (Content Type Diversification)
Deteksi Format yang Kompetitor Punya Tapi Anda Tidak:
| Format | Deteksi | Contoh | Prioritas |
|---|---|---|---|
| Video Tutorial | Competitor YouTube Embed / Video Schema | “Cara Setup n8n” → Video 15 menit | High (Engagement 3x) |
| Interactive Tool/Calculator | Competitor Page dengan Input/Output | “Kalkulator Harga Bot WhatsApp” | High (Link Magnet) |
| Comparison Table | Competitor Table HTML/JS | “n8n vs Zapier vs Make: Tabel Lengkap” | High (Commercial Intent) |
| Downloadable Template | Competitor .xlsx/.pdf/.notion Download | “Template Content Calendar Gratis” | Medium (Lead Gen) |
| Case Study Detail | Competitor “Case Study” / “Testimoni” Section | “Omzet Naik 2x: Studi Kasus Toko X” | High (Trust Signal) |
| PDF/Whitepaper | Competitor .pdf Download | “Panduan Lengkap WhatsApp AI Bot PDF” | Medium (Lead Magnet) |
| Podcast Episode | Competitor Audio Embed / RSS | “Wawancara Founder Toko Sukses” | Low-Medium |
| Webinar/Replay | Competitor Video Embed + Registrasi | “Live Q&A: Setup Bot WhatsApp” | Medium (Engagement) |
4. Freshness Gap Engine (Content Decay vs Competitor Freshness)
- Competitor Freshness Score: Avg Days Since Update (Top 50 Pages) → Lower = Fresher
- Your Content Age: Days Since Last Update per URL
- Gap: Competitor Avg Age < Your Avg Age - 30 hari = Freshness Gap
- Seasonal Freshness: Competitor Update Seasonal (Tahun Baru, Ramadan, Back to School) vs Anda
AI Brief Generator: Dari Gap ke Content Brief Siap Tulis
Prompt: Content Brief Generator (per Gap Item)
Buat Content Brief lengkap untuk mengisi Gap: “{{Gap_Topic}}”\nKompetitor Referensi: {{Competitor_URLs}}\nGap Type: {{Gap_Type}} (MISSING/THIN/OUTDATED/FORMAT)\nGap Score: {{Gap_Score}}\nKeyword Target: {{Target_Keywords}} (Volume: {{Volumes}}, KD: {{KDs}})\nIntent: {{Intent}}\n\nOUTPUT JSON SAJA:\n{\n “content_id”: “CNT-{{YYYYMMDD}}-{{NNN}}”,\n “title_options”: [3 opsi judul: SEO Optimized, Click-worthy, Question-based],\n “target_keyword”: “{{Primary_Keyword}}”,\n “secondary_keywords”: [{{LSI_Keywords}}],\n “search_intent”: “{{Intent}}”,\n “content_format”: “{{Recommended_Format}}”,\n “word_count_target”: {{Target_Words}},\n “outline”: [\n {“h2”: “Judul H2”, “h3”: [“Sub H3”], “word_target”: 300, “key_points”: [“…”], “e_e_a_t_angle”: “Experience: … | Expertise: … | Authority: … | Trust: …”}\n ],\n “faq_section”: [{“question”: “…”, “answer_summary”: “…”}],\n “internal_link_opportunities”: [{“anchor”: “…”, “target_url”: “…”, “context”: “…”}],\n “schema_recommendations”: [“Article”, “FAQ”, “HowTo”, “VideoObject”],\n “monetization_hooks”: [{“type”: “AFFILIATE|LEAD|PRODUCT|AD”, “placement”: “H2_3|Conclusion|Sidebar”, “details”: “…”}],\n “e_e_a_t_checklist”: {“experience”: “Berdasarkan setup 50+ bot…”, “expertise”: “Terminologi: n8n workflow, webhook…”, “authority”: “Author: Agus Wibisono, 50+ bot deployed”, “trust”: “AI Disclosure + Fact-check + Correction Policy”},\n “estimated_traffic”: {{Estimated_Monthly_Traffic}},\n “priority”: “{{HIGH/MEDIUM/LOW}}”,\n “assigned_writer”: “AI/Writer_1”,\n “deadline_days”: 3\n}
Integrasi ke Content Pipeline (End-to-End Automation)
Sheets Tabs yang Terlibat
- Seed_Keywords: Manual Input + Auto-discovery (Trends, GSC, Competitor)
- Competitor_List: Auto-discovered + Manual Add (Domain, DA, Traffic, Niche, Priority)
- Competitor_Gap_Report: Content Gap (Topic Level)
- Keyword_Gap_Report: Keyword Gap (Keyword Level)
- Format_Gap_Report: Format Gap (Content Type Level)
- Freshness_Gap_Report: Freshness Gap (Recency Level)
- Content_Briefs: Generated Briefs (Ready to Write)
- Content_Pipeline: Master Control (Status: PLANNED → WRITING → REVIEW → PUBLISHED)
Flow Otomatis: Gap → Brief → Pipeline → Assign
# n8n Function Node: Gap to Pipeline\n# Input: Gap Item (dari Gap Report)\n# Output: Content_Pipeline Row Data\n\nconst gap = $json;\n\nconst pipelineRow = {\n Content_ID: `CNT-${new Date().toISOString().slice(0,10).replace(/-/g,”)}-${Math.random().toString(36).substr(2,3).toUpperCase()}`,\n Keyword_Utama: gap.target_keyword || gap.keyword,\n Keyword_LSI: gap.secondary_keywords?.join(‘|’) || ”,\n Search_Intent: gap.intent,\n Cluster_Topic: gap.cluster_topic || gap.pillar_topic,\n Target_Word_Count: gap.word_count_target || 2000,\n Competitor_URLs: gap.competitor_urls?.join(‘|’) || ”,\n Status: ‘PLANNED’,\n Outline_JSON: JSON.stringify(gap.outline),\n Meta_Title: gap.title_options?.[0] || ”,\n Meta_Description: gap.meta_description || ”,\n Schema_JSON: JSON.stringify(gap.schema_recommendations),\n Internal_Links: gap.internal_link_opportunities?.map(l => l.target_url).join(‘|’) || ”,\n Monetization: gap.monetization_hooks?.map(m => m.type).join(‘|’) || ”,\n Priority: gap.priority,\n Assigned_To: gap.assigned_writer || ‘AI’,\n Deadline: new Date(Date.now() + (gap.deadline_days || 3)*86400000).toISOString().split(‘T’)[0],\n Gap_Source: `${gap.gap_type}: ${gap.gap_id}`,\n Estimated_Traffic: gap.estimated_traffic || 0\n};\n\nreturn pipelineRow;
Scheduling & Notification (Weekly Rhythm)
| Waktu | Activity | Output | Channel |
|---|---|---|---|
| Senin 05:00 | Cron Trigger: Competitor Research Automation | Gap Reports + Briefs | n8n Execution |
| Senin 06:00 | Telegram Alert: Summary Report | “📊 23 Gap, 12 High Priority. Top: [Topic 1] (5k traffic), [Topic 2] (3k traffic)” | Telegram Group |
| Senin 08:00 | Editor Review: Top 10 Briefs | Approve/Edit/Reject via Telegram Buttons | Telegram Bot |
| Senin 10:00 | Approved → Auto-create Content_Pipeline Rows | PLANNED Articles dengan Deadline | Sheets + Telegram |
| Selasa-Rabu | Writer (AI/Human) Ambil Task → Writing | Content_Pipeline Status → WRITING | Sheets / Notion / Trello |
| Kamis-Jumat | Review & Publish | Content_Pipeline Status → PUBLISHED | WordPress + Sheets |
| Sabtu | Indexing Check + Interlink | IndexNow + Google API + Interlink Engine | n8n Cron |
| Minggu | Weekly Report: Published vs Planned, Gap Closed, Traffic Impact | Dashboard Update + Next Week Preview | Telegram + Email |
Checklist Setup Otomasi Riset Kompetitor Minggu Ini
🔧 SETUP API & INFRASTRUKTUR:
- [ ] DataForSEO / Ahrefs / SEMrush API Credentials → n8n Credentials
- [ ] GSC API (Service Account) untuk Your Keywords Baseline
- [ ] Screaming Frog CLI / Custom Crawler untuk Competitor Content Extraction
- [ ] OpenAI API Key untuk Embedding + AI Analyzers
🤖 BANGUN 4 GAP ENGINES + BRIEF GENERATOR:
- [ ] Content Gap Engine: Topic Extraction → Semantic Match → Gap Scoring
- [ ] Keyword Gap Engine: Organic Keywords API → Gap Identification → AI Enrichment
- [ ] Format Gap Engine: Format Detection → Gap Classification → Recommendation
- [ ] Freshness Gap Engine: Content Age Comparison → Seasonal Analysis
- [ ] Brief Generator: Gap → Outline + Keywords + Structure + E-E-A-T + Monetization
📊 SHEETS & DASHBOARD:
- [ ] Setup 8 Tabs Sheets (Seed, Competitor, 4 Gap Reports, Briefs, Pipeline)
- [ ] Setup Dashboard: Gap Summary, Priority Matrix, Trend Tracking
- [ ] Setup Content_Pipeline Integration: Auto-create Rows dari Approved Briefs
🔔 NOTIFIKASI & WORKFLOW:
- [ ] Telegram Bot: Weekly Summary + Approval Buttons (APPROVE/EDIT/REJECT)
- [ ] Cron Weekly (Senin 05:00) + Manual Trigger (/riset command)
- [ ] Integration Content_Pipeline: Auto-create PLANNED Rows dari Approved Briefs
🧪 TEST & LAUNCH:
- [ ] Test: 3 Competitor → 4 Gap Reports → 10 Briefs → Verify Quality
- [ ] Benchmark vs Manual Research: Accuracy > 90%, Time < 15 menit
- [ ] Launch: Weekly Cron → 4 Minggu Review → Optimize Prompts & Scoring
FAQ: Otomasi Riset Kompetitor & Content Gap dengan n8n + AI
Apakah butuh subscription Ahrefs/SEMrush mahal untuk ini?
OPSIONAL — DataForSEO menyediakan Organic Keywords, Top Pages, Referring Domains, Backlinks API dengan harga pay-per-use (~$0.5-2/1000 request). Jauh lebih murah. Ahrefs/SEMrush hanya untuk validasi atau fitur advance (Site Audit, Link Intersect UI). Start dengan DataForSEO, upgrade saat butuh.
Bagaimana handle kompetitor yang block bot / rate limit API?
1. Rotasi User-Agent & IP (Proxy Rotation Service). 2. Rate Limit Respect: Delay antar request, Exponential Backoff. 3. Cache Result: Simpan ke Sheets/DB, TTL 7 hari (jangan fetch ulang tiap minggu data yang sama). 4. Fallback: Jika API gagal, gunakan DataForSEO (lebih robust) atau manual check top 5 kompetitor. 5. Alert: Competitor block → Alert Owner → Manual check.
Apakah analisis ini legal & etis (white hat)?
YA — Semua data: Public SERP, Public Backlink Data (Ahrefs/SEMrush index public web), Public Technical Metrics (PageSpeed, Schema). Tidak: Hacking, Scraping private area, PII, Copyright infringement. Analisis kompetitor = Standard business intelligence. Google sendiri merekomendasikan analisis kompetitor untuk improve content.
Berapa biaya API riset 20 kompetitor per minggu?
DataForSEO: Organic Keywords (20k) ~$10, Top Pages (20) ~$1, Referring Domains (20k) ~$5, Backlinks (20k) ~$5, PageSpeed (100) ~$0.50 = ~$21.50/minggu (~$86/bln). Screaming Frog CLI: Gratis (local). Total <$100/bln untuk 20 kompetitor full analysis weekly. Jauh lebih murah vs Ahrefs $99/bln + SEMrush $119/bln = $218/bln.
Template otomasi riset kompetitor lengkap sudah jadi di mana?
Ebook + Template n8n — include: Competitor Research Workflow (Discovery → 4 Gap Engines → Brief Generator → Pipeline), 4 AI Prompt Library (Content, Keyword, Format, Freshness Gap), Sheets Template 8 Tabs, Brief Generator Prompt, Pipeline Integration Function, Weekly Cron Setup, Telegram Alert Template, Accuracy Benchmark Guide, Legal Compliance Checklist.
