Member Avatar for MrCapuchino

Hello,

I have a Text file containing the english dictionary, it is 237,000 words long and in each line of the text file there is a word.

What I want to do is obtain a word from the textfile randomly. I don't think reading the text file to end and storing it in an array is an efficient solution. Is there a way I could do to skip reading all the file and just access a random line?

Oh, another thing how can I add the text file to the project so I can use a relative link to the file, since I have to use a complete path in my harddrive and I don't think that will work when I put the project in another computer.

Thanks

Edited by MrCapuchino because: n/a
  • 4 Contributors
  • 5 Replies
  • 2K Views
  • 7 Hours Discussion Span
  • Latest Post Latest Post by MrCapuchino

Dani AI

Generated

If you want uniform randomness without loading 237k lines into memory, use reservoir sampling. It scans the file once, keeps O(1) memory, and you do not need to know the line count in advance. This avoids the list-all-lines approach posted, while still agreeing with @Mitja Bonca that you must read up to the chosen line (a single sequential pass per pick).

import random
from pathlib import Path

def random_word(path: Path) -> str | None:
    pick = None
    with path.open('r', encoding='utf-8') as f:
        for n, line in enumerate(f, 1):
            if random.randrange(n) == 0:  # probability = 1/n
                pick = line.rstrip('\r\n')
    return pick

# file relative to this script
WORDS = Path(__file__).parent / 'words.txt'
print(random_word(WORDS))

If you need many random selections and want them to be fast, build a one-time byte-offset index and reuse it. Roughly, write the file position at the start of each line into a binary .idx file (8 bytes per line; 237k lines ~ 1.9 MB). Then for each pick: choose a random index i, seek(offset[i]), and readline(). That gives O(1) time per selection with minimal memory during picks. It is also portable and works even if line lengths vary.

Two cautions:

  • Do not just seek to a random byte and take the next line; that biases toward longer lines. The index or reservoir methods avoid that.
  • For portability, avoid hard-coded paths. In Python, use Path(__file__).parent / 'words.txt' as above so the code works when copied to another machine or packaged.

Recommended Answers

All 5 Replies

Member Avatar for darkseid 
public static string GetRandomLine(ref string file)
{
    //Generic list for holding the lines
    List<string> lines = new List<string>();

    //Random class to generate our random number
    Random rnd = new Random();

    //Variable to hold our random line number
    int i = 0;

    try
    {
        if (File.Exists(file))
        {
            //StreamReader to read our file
            StreamReader reader = new StreamReader(file);
            
            //Now we loop through each line of our text file
            //adding each line to our list
            while (!(reader.Peek() == -1))
                lines.Add(reader.ReadLine());

            //Now we need a random number
            i = rnd.Next(lines.Count);

            //Close our StreamReader
            reader.Close();

            //Dispose of the instance
            reader.Dispose();

            //Now write out the random line to the TextBox
            return lines[i].Trim();
        }
        else
        {
            //file doesn't exist so return nothing
            return string.Empty;
        }
    }
    catch (IOException ex)
    {
        MessageBox.Show("Error: " + ex.Message);
        return string.Empty;
    }

}

for file path
you put the file in the \debug folder
and use

string filepath = System.IO.Directory.GetCurrentDirectory() + "\\filename";
Edited by darkseid because: n/a
Member Avatar for jugosoft

1. Generate a random number with Random class. There must be an option in StreamReader/TextReader to get block from the specific location.

2. If that text file is in the same folder where application is, use just file's name eg.

StreamReader reader = new StreamReader("file.txt");
Member Avatar for MrCapuchino

Hi,

Thanks for your answer

darkseid, I think that your solution would not work very efficiently with me since the text file is very large and I don't want to waste resources, what I want is not having to store all the text file in memory.

jugosoft, what do you mean with block of specific location can you help me throught?

Thanks

Member Avatar for Mitja Bonca

This is the best possible solution I will give you now that I know.
YOu cannot read only one line from a text file, like read at index. No, you can read some random line, but until that line, the code will read all the lines from 1st to that line you want.

string path = @"C:\1\test5.txt";
            int allRows = File.ReadAllLines(path).Length;
            Random random = new Random();
            int myLine = random.Next(0, allRows);
            using (Stream stream = File.Open(path, FileMode.Open))
            {
                using (StreamReader reader = new StreamReader(stream))
                {
                    string line = null;
                    for (int i = 0; i < myLine; ++i)
                    {
                        line = reader.ReadLine();
                    }
                    Console.WriteLine("Randonly selected text: {0}", line);
                }
                Console.ReadLine();
            }
Member Avatar for MrCapuchino

I understand, it is a nice solution, What I will do is shorten the dictionary erasing the word that I don't need then use your coude.

Thank You

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.