Setup
If ~/hadoop/ doesn't exist or is empty, read setup.md and start the conversation naturally.
When to Use
User works with Hadoop ecosystem (HDFS, YARN, MapReduce, Hive). Agent handles cluster diagnostics, job optimization, storage management, and troubleshooting distributed processing failures.
Architecture
Memory lives in ~/hadoop/. See memory-template.md for structure.
CODEBLOCK0
Quick Reference
| Topic | File |
|---|
| Setup process | INLINECODE4 |
| Memory template |
memory-template.md |
| HDFS operations |
hdfs.md |
| YARN tuning |
yarn.md |
| Troubleshooting |
troubleshooting.md |
Core Rules
1. Verify Cluster State First
Before any operation, check cluster health:
hdfs dfsadmin -report
yarn node -list
Never assume cluster is healthy. A single dead DataNode changes everything.
2. Storage Before Compute
HDFS issues cascade into job failures. Always check:
hdfs dfs -df -h # Capacity
hdfs fsck / -files -blocks # Block health
A job failing with "No space left" is storage, not code.
3. Resource Calculator Awareness
YARN allocates based on configured scheduler. Know which is active:
yarn rmadmin -getServiceState rm1
cat /etc/hadoop/conf/yarn-site.xml | grep scheduler
Default (Capacity) vs Fair scheduler behave very differently.
4. Replication Factor Context
Default replication=3. For temp data, suggest 1-2 to save space:
hdfs dfs -setrep -w 1 /tmp/scratch/
For critical data, verify replication is honored:
CODEBLOCK5
5. Log Location Awareness
Hadoop logs scatter across machines. Key locations:
| Component | Log Path |
|---|
| NameNode | /var/log/hadoop-hdfs/hadoop-hdfs-namenode-.log |
| DataNode |
/var/log/hadoop-hdfs/hadoop-hdfs-datanode-.log |
| ResourceManager | /var/log/hadoop-yarn/yarn-yarn-resourcemanager-*.log |
| NodeManager | /var/log/hadoop-yarn/yarn-yarn-nodemanager-*.log |
| Application | yarn logs -applicationId
|
6. Safe Mode Handling
NameNode enters safe mode on startup or low block count:
hdfs dfsadmin -safemode get # Check status
hdfs dfsadmin -safemode leave # Exit (if blocks OK)
Never force-leave if blocks are actually missing.
7. Memory Settings Matter
90% of "job killed" issues are memory:
# Container settings
yarn.nodemanager.resource.memory-mb # Total per node
yarn.scheduler.minimum-allocation-mb # Min container
mapreduce.map.memory.mb # Map task
mapreduce.reduce.memory.mb # Reduce task
Check these before assuming code is wrong.
HDFS Operations
Essential Commands
CODEBLOCK8
Block Management
CODEBLOCK9
YARN Job Management
Application Lifecycle
CODEBLOCK10
Queue Management
CODEBLOCK11
Common Traps
- - Deleting without -skipTrash on full cluster → Trash still uses space, cluster stays full
- Setting container memory below JVM heap → Instant container kill, confusing errors
- Ignoring speculative execution on slow jobs → Wastes resources on duplicated tasks
- Running fsck on busy cluster → Performance impact, run during maintenance
- Assuming HDFS = POSIX semantics → No append-in-place, no random writes
- Forgetting timezone in scheduling → Oozie/Airflow jobs fire at wrong times
Security & Privacy
Data that stays local:
- - Cluster notes saved in ~/hadoop/clusters/
- Preferences and environment context
What commands access:
- - hdfs/yarn commands connect to your Hadoop cluster
- Some commands read system paths (/var/log, /etc/hadoop/conf)
- Destructive commands require explicit user confirmation
This skill does NOT:
- - Store credentials (use kinit/keytab separately)
- Make external API calls beyond your cluster
- Run destructive commands without asking first
Related Skills
Install with clawhub install <slug> if user confirms:
- -
linux — system administration - INLINECODE11 — containerized deployments
- INLINECODE12 — shell scripting
Feedback
- - If useful: INLINECODE13
- Stay updated: INLINECODE14
设置
如果 ~/hadoop/ 目录不存在或为空，请阅读 setup.md 并自然开始对话。
使用时机
用户使用 Hadoop 生态系统（HDFS、YARN、MapReduce、Hive）时。助手负责集群诊断、作业优化、存储管理以及分布式处理故障排查。
架构
记忆存储在 ~/hadoop/ 目录中。结构请参考 memory-template.md。
~/hadoop/
├── memory.md # 集群配置、常见问题、偏好设置
├── clusters/ # 每个集群的笔记和配置
│ └── {name}.md # 特定集群上下文
└── scripts/ # 自定义诊断脚本
快速参考
memory-template.md |
| HDFS 操作 | hdfs.md |
| YARN 调优 | yarn.md |
| 故障排查 | troubleshooting.md |
核心规则
1. 首先验证集群状态
在执行任何操作前，检查集群健康状态：
bash
hdfs dfsadmin -report
yarn node -list
切勿假设集群健康。单个 DataNode 宕机就会改变一切。
2. 存储优先于计算
HDFS 问题会级联导致作业失败。始终检查：
bash
hdfs dfs -df -h # 容量
hdfs fsck / -files -blocks # 块健康状态
作业因无剩余空间而失败是存储问题，而非代码问题。
3. 资源计算器感知
YARN 根据配置的调度器进行分配。需了解当前激活的调度器：
bash
yarn rmadmin -getServiceState rm1
cat /etc/hadoop/conf/yarn-site.xml | grep scheduler
默认（容量）调度器与公平调度器行为差异很大。
4. 副本因子上下文
默认副本数=3。对于临时数据，建议设为1-2以节省空间：
bash
hdfs dfs -setrep -w 1 /tmp/scratch/
对于关键数据，验证副本是否生效：
bash
hdfs fsck /data/critical -files -blocks -replicaDetails
5. 日志位置感知
Hadoop 日志分散在多台机器上。关键位置：
| 组件 | 日志路径 |
|---|
| NameNode | /var/log/hadoop-hdfs/hadoop-hdfs-namenode-.log |
| DataNode |
/var/log/hadoop-hdfs/hadoop-hdfs-datanode-.log |
| ResourceManager | /var/log/hadoop-yarn/yarn-yarn-resourcemanager-*.log |
| NodeManager | /var/log/hadoop-yarn/yarn-yarn-nodemanager-*.log |
| 应用 | yarn logs -applicationId
|
6. 安全模式处理
NameNode 在启动或块数量不足时进入安全模式：
bash
hdfs dfsadmin -safemode get # 检查状态
hdfs dfsadmin -safemode leave # 退出（如果块正常）
如果块确实缺失，切勿强制退出。
7. 内存设置至关重要
90%的作业被终止问题与内存相关：
bash
容器设置
yarn.nodemanager.resource.memory-mb # 每节点总量
yarn.scheduler.minimum-allocation-mb # 最小容器
mapreduce.map.memory.mb # Map 任务
mapreduce.reduce.memory.mb # Reduce 任务
在假设代码错误前，先检查这些设置。
HDFS 操作
基本命令
bash
导航
hdfs dfs -ls /path
hdfs dfs -du -h /path # 带人类可读单位的大小
hdfs dfs -count -q /path # 配额信息
数据移动
hdfs dfs -put local.txt /hdfs/ # 上传
hdfs dfs -get /hdfs/file.txt . # 下载
hdfs dfs -cp /src /dst # HDFS 内复制
hdfs dfs -mv /src /dst # HDFS 内移动
维护
hdfs dfs -rm -r /path # 删除（移至回收站）
hdfs dfs -rm -r -skipTrash /path # 删除（永久）
hdfs dfs -expunge # 清空回收站
块管理
bash
查找损坏块
hdfs fsck / -list-corruptfileblocks
删除损坏文件（确认无法恢复后）
hdfs fsck /path/file -delete
强制复制
hdfs dfs -setrep -w 3 /important/data/
YARN 作业管理
应用生命周期
bash
列出应用
yarn application -list # 运行中
yarn application -list -appStates ALL # 所有状态
应用详情
yarn application -status
终止卡住的应用
yarn application -kill
获取日志（完成后）
yarn logs -applicationId
yarn logs -applicationId id> -containerId id>
队列管理
bash
列出队列
yarn queue -list
队列状态
yarn queue -status
在队列间移动应用
yarn application -movetoqueue id> -queue queue>
常见陷阱
- - 在集群已满时删除而不使用 -skipTrash → 回收站仍占用空间，集群持续满负荷
- 将容器内存设置为低于 JVM 堆内存 → 容器立即被终止，产生令人困惑的错误
- 在慢作业上忽略推测执行 → 重复任务浪费资源
- 在繁忙集群上运行 fsck → 影响性能，应在维护期间运行
- 假设 HDFS = POSIX 语义 → 不支持原地追加、随机写入
- 调度时忘记时区 → Oozie/Airflow 作业在错误时间触发
安全与隐私
本地保存的数据：
- - 集群笔记保存在 ~/hadoop/clusters/
- 偏好设置和环境上下文
命令访问的内容：
- - hdfs/yarn 命令连接到您的 Hadoop 集群
- 部分命令读取系统路径（/var/log、/etc/hadoop/conf）
- 破坏性命令需要用户明确确认
此技能不会：
- - 存储凭据（请单独使用 kinit/keytab）
- 对集群之外进行外部 API 调用
- 未经询问直接运行破坏性命令
相关技能
如果用户确认，使用 clawhub install 安装：
- - linux — 系统管理
- docker — 容器化部署
- bash — Shell 脚本
反馈
- - 如果觉得有用：clawhub star hadoop
- 保持更新：clawhub sync
