Member Avatar for Niloofar24

Hi.
I'm creating a web crawler, I want to send the output into a txt file, how can I do it? And I also want to give a path to the script to set the directory. How can I do it now?

  • 2 Contributors
  • 1 Reply
  • 389 Views
  • 12 Minutes Discussion Span
  • Latest Post Latest Post by Gribouillis

Dani AI

Generated

Brief recap: wants crawler output written to a text file and a way to pass the destination path; showed the basic write pattern and mentioned interactive/GUI entry. A cleaner, more robust pattern is to accept a path as a command-line argument or config value, normalize and create the directory if missing, then write in a safe, encoded way so the crawler works cross-platform and handles errors gracefully.

A compact, practical pattern (argparse + pathlib) — accept the path, ensure it exists, then save:

import argparse
from pathlib import Path

parser = argparse.ArgumentParser()
parser.add_argument("outdir", help="output directory")
args = parser.parse_args()

outdir = Path(args.outdir).expanduser().resolve()
outdir.mkdir(parents=True, exist_ok=True)

outfile = outdir / "results.txt"
outfile.write_text(results, encoding="utf-8")

Notes and gotchas not covered above: normalize ~ and relative paths with Path.expanduser() / Path.resolve(); create directories safely with mkdir(parents=True, exist_ok=True); prefer Path.write_text() (Python 3) or an atomic write (write to a temp file in the same directory then os.replace() on success) to avoid partial files if the process is interrupted. For multithreaded crawlers, serialize file writes (single writer thread or a thread-safe queue) or use an inter-process lock to avoid corruption. Sanitize or hash URLs before turning them into filenames to avoid illegal characters and collisions. For very large output, stream to the file incrementally rather than building one giant string in memory.

References for the APIs and patterns above: argparse documentation (argparse docs), pathlib (pathlib docs), and safe temporary-file patterns (tempfile docs). For GUI directory pickers (if a GUI is preferred) see tkinter.filedialog ().

Member Avatar for Gribouillis

It is very easy to write the output to a text file

with io.open(textfilename, mode='w', encoding='utf-8') as ofh:
    ofh.write(theoutput)

If you want to enter interactively a directory path to save the files, you have 2 solutions: either read the path in the console with the input() method or use a GUI form to select the directory. If your program is already using a gui module, there is probably already a widget to select a directory in your gui module.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.