Member Avatar for ANVESH_2

test.c

void fun(){
        printf("int main char");
    }

    int main()
    {
        printf("int main int");
    }

Im giving the command like this "run.sh cpd --minimum-tokens 5 --files /opt/test.c --language c" and the output is as follows :

Found a 2 line (5 tokens) duplication in the following files:
Starting at line 1 of /opt/test.c
Starting at line 5 of /opt/test.c

void fun(){
    printf("int main char ");

Even if there is no duplication, it is telling the code is duplicated because of minimum tokens. Is there any way to specify the command without minimum tokens flag.

void fun(){
    printf("int main int");
}

int main(){
    printf("int main int");
}

I specified the command like this "run.sh cpd --minimum-tokens 9 --files /opt/test.c --language c" the output is as follows:

  Added /opt/test.c <--- (No line duplication reported by tool)

This is because I specified the token value greater than the required token value which is 8. In the above case even if there is a duplicated code the tool is not returning any duplicated code.

So how to conclude on Min token size in such scenario in order to get correct duplication of code

Edited by ANVESH_2
  • 3 Contributors
  • 3 Replies
  • 1K Views
  • 1 Week Discussion Span
  • Latest Post Latest Post by ANVESH_2

Dani AI

Generated

CPD looks for duplicated token sequences, not identical lines of text. Very small thresholds (like 5 tokens) will routinely flag common short patterns (calls, punctuation, braces) as "duplicates"; conversely a threshold larger than the token-length of an actual repeated block will miss it. The CPD manual explains that --minimum-tokens is the minimum token length to report and shows example runs using 100 tokens as a typical starting point. See the CPD docs for the option list and examples: CPD user docs. (pmd.github.io)

Practical approach to pick a good --minimum-tokens for a C/C++ tree:

  • Start with a conservative default (100 tokens) and run CPD on the repo to see noise vs. real hits.
  • Identify a handful of known duplicates you want CPD to find, then lower --minimum-tokens until those are detected; stop before common boilerplate (includes, small call sites) becomes noisy. For many C projects small helper functions may need thresholds in the 25–50 range, libraries and larger modules often use 75–100.
  • Use CPD options like --ignore-identifiers and --ignore-literals to detect structural clones that differ only by names or constants. See the CPD options reference for these flags. CPD options reference. (pmd.github.io)

A note on versions and false positives: older PMD/CPD releases (for example v5.4.1) have had language-specific false positives reported; if you see odd behavior test against a newer CPD or a small isolated corpus to verify tokenization. Upgrading CPD can fix lexer issues that cause spurious matches. PMD bug tracker / issue example (5.4.1). (sourceforge.net)

This advice builds on ’s examples (low threshold => noise; too-high threshold => missed clones) and echoes ’s caution that tools must be tuned rather than treated as infallible; compiler version (asked by ) is not what CPD uses for tokenization—CPD uses its own language lexer. (pmd.github.io)

Recommended Answers

All 3 Replies

Member Avatar for rproffitt

I took a look at this old thing and there's a book and more out there. https://pmdapplied.com/

What I see is something that looks to have no active updates for about 5 years so this is a tool and like any tool of this ilk, once in a while you find someone thinking it does "this." A few others may think it's a panacea.

Don't get me wrong I like automated code inspection but there are folk that want to make it a rule or a gauntlet thou shall pass. (or not.) For me I'll get by with turning all the warnings on with a simple -wall.

Member Avatar for rubberman

Honestly, in 35 years of C and C++ software development professionally, I have never seen such an error. What compiler (and version) are you using?

Member Avatar for ANVESH_2

gcc compiler 4.4.7 version.
CPD-PMD version is 5.4.1

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.