Member Avatar for Bug_Catcher

I want to measure the throughput at each datanode by measuring the time taken for each read/write operation. It is very confusing to read through the million functions and find out where this is happening. Could someone list the series of calls made while reading/writing a block of data? am using version 1.0.1. Alternatively, if there is already an API which measures this at the datanode I could use that information.

  • 2 Contributors
  • 2 Replies
  • 108 Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by Bug_Catcher

Dani AI

Generated

Short summary for (and following up on ): for a vanilla Hadoop 1.0.1 cluster you can get reliable per-datanode throughput in three practical ways — passive network capture, the built‑in metrics subsystem, or light instrumentation inside the DataNode. Below is a concise read/write call flow plus hands‑on options you can apply immediately.

High‑level call flow (read and write)

  • Write: client (DFSClient / DFSOutputStream) asks NameNode for block locations, packetizes data and opens a data connection to the first DataNode. The DataNode accepts via the data transfer service (BlockReceiver), writes to the local store, forwards to the next replica in the pipeline, and acks back to the client. When all acks arrive the client commits the block.
  • Read: client asks NameNode for locations, connects to a DataNode on the data port, issues a read request and the DataNode streams the block (BlockSender) from local storage back to the client.

Practical measurement options

  • Network capture (no Hadoop code changes, minimal risk). Default data transfer port in Hadoop 1.x is 50010; DataNode HTTP UI is 50075. Capture and stat the traffic for that port then compute bytes/sec. Example:
    sudo tcpdump -i eth0 -s 0 -w /tmp/dn.pcap port 50010
tshark -r /tmp/dn.pcap -q -z io,stat,1
  • Metrics subsystem (vanilla Hadoop). Hadoop 1.x has the metrics framework (hadoop-metrics.properties). Enable a CSV or JMX sink on each DataNode to collect bytes read/written and operation counts over time for long‑term monitoring.
  • Instrumentation (per‑block timing). Add small timers in the DataNode transfer classes (BlockReceiver/BlockSender) to record start/end and compute MB/s. Example timing snippet:
    long start = System.nanoTime();
// transfer code
long elapsedNs = System.nanoTime() - start;
double seconds = elapsedNs / 1e9;
double mbps = bytesTransferred / (1024.0*1024.0) / seconds;
LOG.info("block {}: {} bytes in {} ms => {} MB/s", blockId, bytesTransferred, elapsedNs/1e6, mbps);

Notes and cautions

  • Network capture is safest for production diagnostics. Instrumenting transfer code gives per‑block granularity but add careful logging (or sampling) to avoid I/O overhead. Metrics/CVS is best for continuous monitoring.

Recommended Answers

All 2 Replies

Member Avatar for rubberman

The time for any specific read/write function in a hadoop cluster and data node can vary significantly. I don't suppose you are running an industrial strength management tool like Cloudera on your cluster, are you? They do track those sort of metrics, and can alert you when they exceed specified limits.

Member Avatar for Bug_Catcher

No. I cannot assume any metric logging system like Ganglia. This must work on a "vanilla" Hadoop distribution

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.