Skroller - Social Media Content collection
Automate the collection and analysis of publicly available social media content. This skill handles content collection, filtering, and export with compliance safeguards.
⚖️ Legal & Compliance Requirements
Before using this skill:
- 1. Review Platform ToS - Each platform has different rules about automated access
- Check robots.txt - Respect disallowed paths
- Rate Limiting - Stay within platform rate limits to avoid service disruption
- Privacy Laws - GDPR, CCPA, and other regulations apply to personal data
- Permitted Use - Research, personal analysis, competitive intelligence (where allowed)
- Prohibited Use - Spam, harassment, data resale, manipulation, bypassing auth
Data Protection:
- - Anonymize personal data when storing
- Honor deletion requests (GDPR Art. 17)
- Limit retention to necessary periods
- Document lawful basis for processing
- Do not scrape sensitive personal data
Core Capabilities
- - Content collection - Gather publicly available posts from platforms
- Data extraction - Pull text, timestamps, engagement metrics, author info
- Smart filtering - Filter by keywords, hashtags, date ranges, engagement thresholds
- Deduplication - Track seen posts to avoid duplicates across sessions
- Export formats - JSON, CSV, Markdown, or direct to note apps (Bear/Obsidian)
- Rate limiting - Respect platform constraints and avoid service disruption
Platform Support
| Platform | Approach | Notes |
|---|
| Twitter/X | Browser automation | Use Playwright; handle login if needed |
| Instagram |
Browser automation | Rate-limited; use sparingly |
| TikTok | Browser automation | Heavy JS; may need longer waits |
| Reddit | API + browser | Prefer API where possible |
| LinkedIn | Browser automation | Login required for most content |
| YouTube | API + browser | Comments via browser, videos via API |
| Product Hunt | Browser automation | Product discovery, launches |
| Medium | Browser automation | Articles, blog posts |
| GitHub | Browser automation | Issues, discussions, repos |
| Pinterest | Browser automation | Visual content, pins |
Quick Start
Basic Scroll and Extract
CODEBLOCK0
Monitor Brand Mentions
CODEBLOCK1
Competitive Research
CODEBLOCK2
Scripts
scripts/skroller.js - Main scrolling engine
JavaScript script using Playwright for browser automation.
CODEBLOCK3
Options:
- -
--platform - Target platform (required): twitter, reddit, instagram, tiktok, linkedin, youtube, producthunt, medium, github, pinterest - INLINECODE2 - Search keyword/hashtag
- INLINECODE3 - Specific profile to scroll
- INLINECODE4 - Max posts to scrape (default: 50)
- INLINECODE5 - Filter by minimum engagement
- INLINECODE6 - Output format: json, csv, markdown (default: json)
- INLINECODE7 - Output file path
- INLINECODE8 - Capture screenshots for debugging
- INLINECODE9 - Skip previously seen posts
scripts/feed-digest.js - Generate digests
Creates summary digests from exported post data.
CODEBLOCK4
scripts/export-to-notes.js - Unified note app exporter
Exports scraped posts to multiple note applications with a single command.
Supported apps: Bear, Obsidian, Notion, Apple Notes, Evernote, OneNote, Google Keep, Roam Research, Logseq, Anytype
CODEBLOCK5
Configuration: Set defaults in .skroller-config.json:
CODEBLOCK6
Requirements by app:
- - Bear: grizzly CLI (
go install github.com/tylerwince/grizzly/cmd/grizzly@latest) - Obsidian: Vault path
- Notion: API key (create at notion.so)
- Apple Notes: macOS with Notes app
- Evernote: Manual ENEX import
- OneNote: Microsoft Graph access token
- Google Keep: Manual HTML import
- Roam Research: Markdown import (drag MDL file into Roam)
- Logseq: Vault path (writes to pages/ folder)
- Anytype: Markdown import (use Anytype Import feature)
Configuration
.skroller-config.json
Store default settings:
CODEBLOCK7
Authentication
Some platforms require login. Store credentials securely:
CODEBLOCK8
Security note: Never commit auth files. Use .env with .gitignore.
Output Structure
JSON Output (default)
CODEBLOCK9
Markdown Output
CODEBLOCK10
Filtering Strategies
Keyword Matching
- - Exact match: INLINECODE17
- Boolean: INLINECODE18
- Exclude: INLINECODE19
Engagement Thresholds
- - Filter low-quality: INLINECODE20
- Viral content: INLINECODE21
Time Windows
- - Recent only: INLINECODE22
- Historical: INLINECODE23
Best Practices
Rate Limiting
- - Add delays between scrolls: INLINECODE24
- Respect platform limits (see config)
- Use proxies for high-volume scraping
Content Quality
- - Filter by engagement to find signal
- Dedupe across sessions
- Export with timestamps for freshness tracking
Ethics & Compliance
- - Check platform ToS before scraping
- Don't scrape personal data at scale
- Use for research/curation, not spam
Troubleshooting
Scroll stops early
- - Increase
--limit or check for login requirements - Some platforms detect automation; add random delays
Missing content
- - Some platforms lazy-load; increase scroll delay
- Try
--screenshot to debug what's visible
Rate limited
- - Reduce frequency; use INLINECODE27
- Check platform-specific rate limits in config
Integration with Other Skills
- - bear-notes: Export via INLINECODE28
- obsidian: Export via INLINECODE29
- notion: Export via INLINECODE30
- github: Create issues from scraped feedback/mentions
CODEBLOCK11
See Also
- -
references/platform-details.md - Platform-specific selectors and quirks - INLINECODE32 - Rate limit guidelines per platform
- INLINECODE33 - CSS selectors for each platform
Skroller - 社交媒体内容收集
自动化收集和分析公开可用的社交媒体内容。该技能负责内容收集、过滤和导出，并内置合规保障措施。
⚖️ 法律与合规要求
使用此技能前：
- 1. 审查平台服务条款 - 每个平台对自动化访问有不同的规定
- 检查 robots.txt - 遵守禁止访问的路径
- 速率限制 - 保持在平台速率限制内，避免服务中断
- 隐私法律 - GDPR、CCPA 及其他法规适用于个人数据
- 允许用途 - 研究、个人分析、竞争情报（在允许范围内）
- 禁止用途 - 垃圾信息、骚扰、数据转售、操纵、绕过认证
数据保护：
- - 存储时对个人数据进行匿名化处理
- 尊重删除请求（GDPR 第 17 条）
- 将保留期限限制在必要范围内
- 记录处理的法律依据
- 不抓取敏感个人数据
核心功能
- - 内容收集 - 从平台收集公开帖子
- 数据提取 - 提取文本、时间戳、互动指标、作者信息
- 智能过滤 - 按关键词、话题标签、日期范围、互动阈值过滤
- 去重 - 追踪已查看帖子，避免跨会话重复
- 导出格式 - JSON、CSV、Markdown，或直接导出到笔记应用（Bear/Obsidian）
- 速率限制 - 尊重平台限制，避免服务中断
平台支持
| 平台 | 方法 | 备注 |
|---|
| Twitter/X | 浏览器自动化 | 使用 Playwright；必要时处理登录 |
| Instagram |
浏览器自动化 | 有速率限制；谨慎使用 |
| TikTok | 浏览器自动化 | JS 较重；可能需要更长的等待时间 |
| Reddit | API + 浏览器 | 尽可能优先使用 API |
| LinkedIn | 浏览器自动化 | 大多数内容需要登录 |
| YouTube | API + 浏览器 | 评论通过浏览器，视频通过 API |
| Product Hunt | 浏览器自动化 | 产品发现与发布 |
| Medium | 浏览器自动化 | 文章、博客帖子 |
| GitHub | 浏览器自动化 | 问题、讨论、仓库 |
| Pinterest | 浏览器自动化 | 视觉内容、图钉 |
快速开始
基本滚动与提取
bash
滚动 Twitter 信息流，提取 50 条关于 AI 的帖子
node scripts/skroller.js --platform twitter --query AI --limit 50 --output posts.json
监控品牌提及
bash
监控 Reddit 上的品牌提及，导出为 Markdown
node scripts/skroller.js --platform reddit --query mybrand --format markdown --output mentions.md
竞争研究
bash
滚动竞争对手的 Instagram，捕获按互动量排序的热门帖子
node scripts/skroller.js --platform instagram --profile @competitor --min-likes 1000 --output competitor-posts.json
脚本
scripts/skroller.js - 主滚动引擎
使用 Playwright 进行浏览器自动化的 JavaScript 脚本。
bash
使用 npm 脚本（推荐）
npm run scroll -- --platform twitter --query AI --limit 50 --output posts.json
直接执行
node scripts/skroller.js --platform twitter --query AI --limit 50 --output posts.json
选项：
- - --platform - 目标平台（必填）：twitter、reddit、instagram、tiktok、linkedin、youtube、producthunt、medium、github、pinterest
- --query - 搜索关键词/话题标签
- --profile - 要滚动的特定个人资料
- --limit - 最大抓取帖子数（默认：50）
- --min-likes - 按最低互动量过滤
- --format - 输出格式：json、csv、markdown（默认：json）
- --output - 输出文件路径
- --screenshot - 捕获截图用于调试
- --dedupe - 跳过之前已查看的帖子
scripts/feed-digest.js - 生成摘要
从导出的帖子数据创建摘要。
bash
npm run digest -- --input posts.json --output digest.md
或：node scripts/feed-digest.js --input posts.json --output digest.md
scripts/export-to-notes.js - 统一笔记应用导出器
通过单个命令将抓取的帖子导出到多个笔记应用。
支持的应用： Bear、Obsidian、Notion、Apple Notes、Evernote、OneNote、Google Keep、Roam Research、Logseq、Anytype
bash
使用 npm 脚本（推荐）
npm run export -- --input posts.json --app obsidian --vault ~/Documents/Obsidian
直接执行
node scripts/export-to-notes.js --input posts.json --app bear --tags ai,research
node scripts/export-to-notes.js --input posts.json --app notion --api-key $NOTION_TOKEN
node scripts/export-to-notes.js --input posts.json --app apple-notes
node scripts/export-to-notes.js --input posts.json --app evernote --output export.enex
node scripts/export-to-notes.js --input posts.json --app one-note --access-token $MS_TOKEN
node scripts/export-to-notes.js --input posts.json --app keep --output keep.html
node scripts/export-to-notes.js --input posts.json --app roam --output roam.md
node scripts/export-to-notes.js --input posts.json --app logseq --vault ~/Documents/Logseq
node scripts/export-to-notes.js --input posts.json --app anytype --output anytype.md
node scripts/export-to-notes.js --input posts.json --app obsidian --dry-run
配置： 在 .skroller-config.json 中设置默认值：
json
{
export: {
defaultApp: obsidian,
vault: ~/Documents/Obsidian,
folder: Skroller,
notionDatabaseId:
}
}
各应用要求：
- - Bear： grizzly CLI（go install github.com/tylerwince/grizzly/cmd/grizzly@latest）
- Obsidian： 仓库路径
- Notion： API 密钥（在 notion.so 创建）
- Apple Notes： 装有 Notes 应用的 macOS
- Evernote： 手动 ENEX 导入
- OneNote： Microsoft Graph 访问令牌
- Google Keep： 手动 HTML 导入
- Roam Research： Markdown 导入（将 MDL 文件拖入 Roam）
- Logseq： 仓库路径（写入 pages/ 文件夹）
- Anytype： Markdown 导入（使用 Anytype 导入功能）
配置
.skroller-config.json
存储默认设置：
json
{
defaultLimit: 50,
scrollDelayMs: 1500,
userAgent: Mozilla/5.0 ...,
platforms: {
twitter: {
loginRequired: false,
rateLimit: 100 次请求/小时
},
instagram: {
loginRequired: true,
rateLimit: 50 次请求/小时
}
},
export: {
defaultFormat: json,
includeImages: true,
includeMetrics: true
}
}
认证
某些平台需要登录。安全存储凭据：
bash
对于需要认证的平台，设置环境变量
export SKROLLR
TWITTERCOOKIE=<认证 cookie>
export SKROLLR
INSTAGRAMUSER=<用户名>
export SKROLLR
INSTAGRAMPASS=<密码>
安全提示： 切勿提交认证文件。使用 .env 并配合 .gitignore。
输出结构
JSON 输出（默认）
json
{
platform: twitter,
query: AI,
scrapedAt: 2026-03-14T20:30:00Z,
posts: [
{
id: 1234567890,
author: @username,
text: 帖子内容...,
timestamp: 2026-03-14T18:00:00Z,
likes: 150,
retweets: 42,
replies: 12,
url: https://twitter.com/...,
media: [image1.jpg],
hashtags: [#AI, #ML]
}
]
}
Markdown 输出
markdown
Twitter 帖子：AI（2026-03-14）
@username - 150 个赞
帖子内容...
