Respect Creative Work
- - Design & text copying: Avoid copying design elements or substantial portions of text; while facts and data aren't typically protected by copyright, their presentation (website layouts, specific text, compilations) often is.
- Source attribution: Properly attribute sources when appropriate; this shows integrity and builds trust with both content creators and your own audience.
- Creator impact: Consider how your use might impact the original creator's work; respecting copyrighted material demonstrates ethical conduct.
Pre-Extraction Verification Steps
I. Access Authorization — Retrieve {domain}/robots.txt and review /terms or /tos endpoints. Proceed only if neither prohibits extraction; halt if blocked or explicit restrictions exist.
II. Data Classification — Distinguish between public factual information (listings, pricing) versus personal information. The latter invokes GDPR/CCPA obligations and requires stronger justification.
III. Preferred Channels — Check whether the platform offers an API. If available, use it instead of direct extraction. Never access content requiring authentication without proper credentials.
Operational Conduct & Compliance
- - Request discipline: Throttle at 2-3 seconds minimum, honor 429 with progressive backoff, maintain connection pooling, and use authentic User-Agent with contact email.
- Access boundaries: robots.txt disregard carries uncertain legal standing (Meta v. Bright Data 2024); publicly accessible content is typically permissible (hiQ v. LinkedIn 2022); circumventing access controls risks CFAA exposure (Van Buren v. US 2021).
- Data & content restrictions: Personal information without permission triggers GDPR/CCPA breach; redistributing copyrighted material constitutes copyright violation.
Information Stewardship
- - PII & profiling restrictions: Remove personal information promptly and avoid correlating data to identify individuals.
- Limit retention: Store only necessary data, purge the rest.
- Activity logging: Record extraction events (what, when, source) to demonstrate responsible conduct if questioned.
Implementation patterns and robots.txt evaluation logic in INLINECODE3
尊重创意作品
- - 设计与文本复制：避免复制设计元素或大量文本内容；虽然事实和数据通常不受版权保护，但其呈现方式（网站布局、特定文本、汇编内容）往往受版权保护。
- 来源标注：在适当情况下正确标注来源；这体现了诚信，有助于与内容创作者及自身受众建立信任。
- 创作者影响：考虑你的使用行为可能对原创作者作品产生的影响；尊重受版权保护的材料是道德行为的体现。
提取前验证步骤
I. 访问授权 — 获取 {domain}/robots.txt 并审查 /terms 或 /tos 端点。仅当这些文件未禁止提取时才继续操作；若被阻止或存在明确限制则立即停止。
II. 数据分类 — 区分公开事实信息（列表、定价）与个人信息。后者涉及GDPR/CCPA合规义务，需要更充分的正当理由。
III. 优先渠道 — 检查平台是否提供API。若可用，应优先使用API而非直接提取。未经有效凭证，切勿访问需要身份验证的内容。
操作规范与合规要求
- - 请求纪律：请求间隔至少2-3秒，遇到429状态码时逐步退避，维护连接池，使用包含联系邮箱的真实用户代理。
- 访问边界：无视robots.txt可能面临法律风险（Meta诉Bright Data案，2024年）；公开可访问内容通常允许提取（hiQ诉LinkedIn案，2022年）；规避访问控制可能触犯CFAA（Van Buren诉美国案，2021年）。
- 数据与内容限制：未经许可处理个人信息将触发GDPR/CCPA违规；重新分发受版权保护的材料构成版权侵权。
信息管理责任
- - PII与画像限制：及时移除个人信息，避免关联数据以识别个人身份。
- 限制留存：仅存储必要数据，其余数据予以清除。
- 活动日志：记录提取事件（内容、时间、来源），以便在受到质疑时证明行为的合规性。
实现模式及robots.txt评估逻辑详见code.md。
