Member Avatar for FrankHere

Hi all,

I am a final year student of a Computing Degree and have an idea of doing a Distributed File System for the Final Year Project. I am mainly consider only Distribution of files across a LAN and Replication of data. So any idea of how to split the files (Bite Streams) into chunks or any valuable research paper will be highly appreciated..

Thanking you..

Rukshan Silva

  • 3 Contributors
  • 2 Replies
  • 107 Views
  • 10 Hours Discussion Span
  • Latest Post Latest Post by jwenting

Dani AI

Generated

As and noted, this should be your own work — below is a compact, practical roadmap and vetted reading to help design and implement a LAN-focused distributed file system with chunking and replication.

Start with a simple, well-known architecture: a single metadata server that stores namespace and chunk-to-server maps, many chunk servers that store fixed-size blocks, and clients that ask metadata for locations then stream directly to chunk servers. Fixed-size blocks are simplest to implement; GFS used 64 MB and HDFS uses 64–128 MB as defaults, but for lots of small files use much smaller blocks (e.g., 4–16 KB to avoid wasted space). See the GFS paper and HDFS design for rationale and tradeoffs (GFS paper, HDFS design).

If deduplication or delta-transfer is wanted, use content-defined chunking (Rabin fingerprinting) and rolling checksums; the rsync technical report and Rabin fingerprint overview explain the algorithms used in practice (rsync tech report, Rabin fingerprint). Always store a checksum per chunk (CRC32 or SHA-1/256) and a version or epoch to detect stale replicas.

Java implementation notes: use java.nio FileChannel / ByteBuffer for streaming and zero-copy where possible, compute checksums as you write, persist metadata (in-memory map + write-ahead log + periodic snapshot), and run a background re-replication thread to heal lost replicas. For coordination or leader election consider ZooKeeper for reliability. Test early: simulate node failures, measure rebuild time, tune chunk size, and watch JVM GC behavior when serving many concurrent transfers.

Recommended reading: GFS and HDFS (architecture/replication), the rsync report and Rabin fingerprinting (chunking/dedup). Keep scope small: implement core read/write/replicate/heal first, then add dedupe or rack-aware placement.

Recommended Answers

All 2 Replies

Member Avatar for Ezzaral

This is your final project to get a degree. It is expected to demonstrate that you have learned enough to warrant that degree. Asking how to do that project calls into question whether you actually have acquired sufficient knowledge and the ability to research and work through the tasks you undertake. There is a level of self-direction necessary to work in the field beyond asking how to do everything on a forum won't cut it in the real world.

I say that merely as advice to you - not to be mean-spirited about it. You really need to work through this project on your own to prepare for employment in any development capacity.

Good luck!

Member Avatar for jwenting

worse, turning in someone elses work as your own is plagiarism (and no teacher is going to accept something you say openly you didn't write yourself) which is enough to get you an automatic failure and exclusion from any form of higher education.

Do you own homework, kiddo. There are enough people out there with fake degrees, we're not going to help someone else get one.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.