Member Avatar for noraz

Hi,

I'm building a small information retrieval (IR) application using VB6. I'm stuck at calculating the tf.idf part. I've google for the solutions/codes, but didn't find anything in vb6, except other languages such as phyton etc (which I'm not familiar with). I'm also a beginner programmer in vb6. I've looked at VB6 Stuff and Tricks Thread (http://www.daniweb.com/forums/thread214396.html), but so far couldn't find the answer here and elsewhere.

If someone could show me the sample code or have knowledge on such links that provided the solution, please let me know. Need help here. Thanks in advance.

  • 2 Contributors
  • 4 Replies
  • 577 Views
  • 3 Days Discussion Span
  • Latest Post Latest Post by noraz

Dani AI

Generated

Building on 's stemming output and 's suggestion to use file I/O, here is a compact, practical VB6 approach that computes per-term TF, per-term DF, IDF, and TF·IDF and writes results to a CSV. Assumes the stemmer output is a simple CSV with one token per line like: term,docID.

First-pass (build counts)

  • Read the CSV with FileSystemObject.
  • Maintain a nested dictionary: term -> (docID -> rawCount).
  • Track a set of unique docIDs to get totalDocs.

Second-pass (compute scores)

  • For each term: df = number of docs in its inner dictionary.
  • idf = Log(totalDocs / df) ' VB6 Log is natural log; for base-10 use Log(x)/Log(10)
  • Choose tf weighting: raw count, normalized (tf/max_tf_in_doc), or log-scale (1 + Log(tf)).
  • tfidf = tf_weight * idf. Optionally L2-normalize each document vector for cosine ranking.

Example VB6 snippet (minimal, two-pass, writes tfidf.csv):

Dim fso As Object, ts As Object, out As Object
Set fso = CreateObject("Scripting.FileSystemObject")
Set ts = fso.OpenTextFile("stemming.csv", 1)

Dim termDict As Object, docs As Object
Set termDict = CreateObject("Scripting.Dictionary")
Set docs = CreateObject("Scripting.Dictionary")

Do While Not ts.AtEndOfStream
    line = ts.ReadLine
    parts = Split(line, ",")
    term = LCase(Trim(parts(0)))
    doc = Trim(parts(1))
    If Not termDict.Exists(term) Then
        Set t = CreateObject("Scripting.Dictionary")
        termDict.Add term, t
    Else
        Set t = termDict(term)
    End If
    If t.Exists(doc) Then t(doc) = t(doc) + 1 Else t.Add doc, 1
    If Not docs.Exists(doc) Then docs.Add doc, True
Loop
ts.Close

totalDocs = docs.Count
Set out = fso.CreateTextFile("tfidf.csv", True)
out.WriteLine "term,doc,tfidf"

For Each term In termDict.Keys
    Set t = termDict(term)
    df = t.Count
    idf = Log(totalDocs / df)
    For Each doc In t.Keys
        tf = t(doc)
        tfw = 1 + Log(tf)      ' example: log-scaled tf
        tfidf = tfw * idf
        out.WriteLine term & "," & doc & "," & CStr(tfidf)
    Next
Next
out.Close

Notes and cautions

  • Add-one smoothing for IDF: idf = Log((totalDocs+1)/(df+1)) to avoid extreme weights.
  • For large corpora, avoid keeping everything in memory: stream counts to a DB or build an inverted index on disk.
  • Remove stopwords, normalize case (done above), and keep stemming consistent.
  • VB6 Log is natural log; convert if a different base is required.

This pattern keeps the implementation simple and debuggable for a beginner VB6 project while allowing you to swap tf weighting and normalization strategies for better ranking.

Recommended Answers

All 4 Replies

Member Avatar for AndreRet

If you can tell us a bit more about tf/idf, what it does, we might be able to supply some code. I had a quick look at some explanations and it did not make much sense to me.

What exactly would you like the app to do?

Member Avatar for noraz

TF referring to Term Frequency and IDF is Inverse Document Frequency. I'm using these later to rank documents.

I have a collection of text documents. I have indexed all the terms in that documents (by applying tokenizing and stemmer). In the stemming output, it will has list of these terms together with their document id. The output of the stemming is a text file with comma delimited.

What I have to do is to calculate for each term in the list, how many does it appears in a document.

While IDF is done by first dividing the total number of documents by the number of documents that contains the actual keyword in question. Then taking the log of the result. Let say, in 10 docs that I have, only 3 docs contained the word "computer". The calculation should be log (10/3).

I'm really not good in converting mathematical formula into source code.

Member Avatar for AndreRet

It makes a bit more sense now. You will probably have to start using file system objects/functions. Just search Daniweb, there are plenty of sample codes.

The following link is a full on tutorial with sample code. I'm sure this will give you the solution you need to read from your files, get the required data and save it to another file called say MyLogs.txt etc.

http://www.vb6.us/tutorials/using-fso-file-system-object-vb6

Member Avatar for noraz

will try on that..
thanx!

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.