How can i creat e text file to use as a database in python

[Niloofar24]

Hi everybody.
How can i creat e text file to use as a database in python?
I want to save my project's data changes into hard disk. What should i do?

[sneekula]

Give us a short example of what your project data looks like.

[Traevel]

If you want to use it as a database, have you considered using sqlite?

From what I gather its usage is fairly straightforward in Python.

[Niloofar24]

I have a file with the name message.py and i have created some functions in it. I have created a list and some functions to be able to append, sort or remove names to my list.
Now i want to save data into my hard disk in order to keep changes. For example i want to append a name to the list and close the program and next time i want changes to be saved to access them again.
I just know that i should use a text file as database.
Now what should i do?

[Gribouillis]

Being able to save data to disk and retrieve it is equivalent to being able to convert this data to string and then back to data. Indeed, you can always define

def save(data, filename):
    with open(filename, 'w') as fh:
        fh.write(string_from_data(data))

def retrieve(filename):
    with open(filename) as fh:
        return data_from_string(fh.read())

It means that you only need to define string_from_data() and data_from_string(). This is called serializing and de-serializing data. There are many ways to do it.

For example, supposing that your data is a list of names and that the names don't contain a newline character, one can define

def string_from_data(data):
    return '\n'.join(data)

def data_from_string(s):
    return s.split('\n')

Now with these definitions, you can write in program:

import os
FILENAME = 'mylist.txt'

if os.path.isfile(FILENAME):
    mylist = retrieve(FILENAME)
else:
    mylist = []

# ... work with mylist

save(mylist, FILENAME)

[vegaseat]

Module pickle is made for this ...

''' pickle_list101.py
use module pickle to dump and load a list object
use binary file modes "wb" and "rb" to make it work with
Python2 and Python3
'''

import pickle

mylist = [
'Frank',
'Paul',
'Mary']

fname = "list101.pkl"
# pickle dump the list object
with open(fname, "wb") as fout:
    # default protocol is zero
    # -1 gives highest prototcol and smallest data file size
    pickle.dump(mylist, fout, protocol=-1)

# pickle load the list object back in (senses protocol)
with open(fname, "rb") as fin:
    mylist2 = pickle.load(fin)

# visual compare
print(mylist)
print('-'*30)
print(mylist2)

''' result ...
['Frank', 'Paul', 'Mary']
------------------------------
['Frank', 'Paul', 'Mary']
'''

[Niloofar24]

Well, where should i save these codes? In the main file? Or in an other file with any special name.txt ?

[vegaseat]

You can store it in any folder you want, as long as you give it the full path name. For convenience, I would keep the code .py file and the data file in one folder.

[snippsat]

A test with dataset,database as simple as it get.
So here i have a player list that i store in a Sqlite(comes with Python) database.

import dataset

player_lst = [
    {'name': 'Tom', 'score': 250},
    {'name': 'Kent', 'score': 150},
    {'name': 'Paul', 'score': 500}
    ]

db = dataset.connect('sqlite:///mydatabase.db')
table = db['Score_table']
for item in player_lst:
    table.insert(item)

Test it out,by taking out some data.

>>> users = db['Score_table'].all()
>>> for user in db['Score_table']:
...     print user
...     
OrderedDict([(u'id', 1), (u'score', 250), (u'name', u'Tom')])
OrderedDict([(u'id', 2), (u'score', 150), (u'name', u'Kent')])
OrderedDict([(u'id', 3), (u'score', 500), (u'name', u'Paul')])

>>> table.find_one(name='Tom')
OrderedDict([(u'id', 1), (u'score', 250), (u'name', u'Tom')])
>>> table.find_one(name='Tom')['score']
250

It also have all the power of SQL queries if needed.

>>> [i for i in db.query("SELECT MAX(score) FROM Score_table")]
[OrderedDict([(u'MAX(score)', 500)])]
>>> [i for i in db.query("SELECT MAX(score) FROM Score_table")][0]['MAX(score)']
500

[Gribouillis]

@snippsat I like the concept, although being based on top of sqlalchemy may have a performance cost for dataset.

The greatest thing in this may be the datafreeze command that it contains and the concept of yaml freezefile which make it possible to describe a reusable extraction of data from a given database.

With a little more work, freezefiles could be used to export pickle files instead of json or csv, making them trivial to load from python.

[snippsat]

@snippsat I like the concept, although being based on top of sqlalchemy may >have a performance cost for dataset.

Yes,i agree there may be some performance cost,
but for many small project where you just want a easy way to get data in and out,i think it can be ok.

With a little more work, freezefiles could be used to export pickle files >instead of json or csv, making them trivial to load from python.

Yes i was looking a little into this freezefiles option.
I like json much more than pickle,and json is in the standard library.
So no need to mix in pickle.

Here is a test with freezefiles.
So here i make json file Score_table.json.

result = db['Score_table'].all()
dataset.freeze(result, format='json', filename='Score_table.json')

Load Score_table.json out,and exctact som data.

import json

with open("Score_table.json") as j:
    score_table = json.load(j)

Test:

>>> score_table
{u'count': -1,
 u'meta': {},
 u'results': [{u'id': 1, u'name': u'Tom', u'score': 250},
              {u'id': 2, u'name': u'Kent', u'score': 150},
              {u'id': 3, u'name': u'Paul', u'score': 500}]}

>>> score_table['results'][0]
{u'id': 1, u'name': u'Tom', u'score': 250}
>>> score_table['results'][1].keys()
[u'score', u'id', u'name']
>>> score_table['results'][1].values()
[150, 2, u'Kent']

>>> print('{} has the highest score of: {}'.format(score_table['results'][2]['name'], score_table['results'][2]['score']))
Paul has the highest score of: 500

export one JSON file per user
You can create one file per row by setting mode to “item”:

Test this out.

result = db['Score_table'].all()
dataset.freeze(result, format='json', filename='users/{{name}}.json', mode='item')

So this create a folder users with 3 json files.
Tom.json Kent.json Paul.json
*****************************************
Here same as vegaseat pickle soultion,but with json.
As i mention i do prefer json over pickle.
And json give you a file on disk that look the same as content in record,
where pickle make no sense before you load it out.

import json

record = {"hans": "2010-01-01", 'grete': "2014-01-01"}
with open("my_file.txt", "w") as f_out:
    json.dump(record, f_out)
with open("my_file.txt") as f:
    saved_record = json.load(f)

print(saved_record)
#--> {u'hans': u'2010-01-01', u'grete': u'2014-01-01'}

[Lardmeister]

With jason
ValueError: Expecting property name enclosed in double quotes
is a royal pain, if you want to save string objects!

[snippsat]

With jason
ValueError: Expecting property name enclosed in double quotes
is a royal pain, if you want to save string objects!

Can you give an example?
No problem in me simple test here.

import json

record = "hello world" + ' test'
with open("my_file.json", "w") as f_out:
    json.dump(record, f_out)
with open("my_file.json") as f:
    saved_record = json.load(f)

print(saved_record)
print(repr(saved_record))
print(type(saved_record))

"""Output-->
hello world test
u'hello world test'
<type 'unicode'>
"""