commenting system and pagination issue : commenting with filw system storage

I think you'll have to cache the number of files in each folder. Otherwise you'll have to recurse through the folders each time counting the number of files each time you call the pagination script.

How are you doing it at the moment?

Thanks ether for post and query.

Actually I am be caching it. We have a SQUID architecture and above all i am using memcache for caching. So i think it is fine.

The concern of 1 system files list call and 2 file read system calls for each comment display in a list of comments is well addressed with caching.

The concern is that on click of say '2' in pagination bar how would i make out that i have to start from file X.X.N in folder A to X.Y in folder B.

This is where I am stuck and worried because after developing the whole API this is where i am left like lost planet.

Yesterday i thought in terms of Btrees or something that sort so that i go through all and then reduce the system calls for reading to painated refered pages clicked. something like this ... it seems like i would go mad now... :(

some pointers and help appreciated.

Thanks ether for post and query.

Actually I am be caching it. We have a SQUID architecture and above all i am using memcache for caching. So i think it is fine.

The concern of 1 system files list call and 2 file read system calls for each comment display in a list of comments is well addressed with caching.

The concern is that on click of say '2' in pagination bar how would i make out that i have to start from file X.X.N in folder A to X.Y in folder B.

This is where I am stuck and worried because after developing the whole API this is where i am left like lost planet.

Yesterday i thought in terms of Btrees or something that sort so that i go through all and then reduce the system calls for reading to painated refered pages clicked. something like this ... it seems like i would go mad now... :(

some pointers and help appreciated.

If you cache the number or files in each folder like: (using your example)

Article
{Folder 1}
Reply 1
Reply to 1 as 1.n
reply to 1.n as 1.n.n
......
{Folder 2}
Reply 2
{Folder 3}
Reply 3

Say you cache an array.

$article1_counts = Array('folder1'=>3, 'folder2'=>1, 'folder3'=>3);

say your limit is 5 (to make the example work).

Then 1 would be folder1 and folder2 and 1 file from folder 3.
Then 2 would be 2 files from folder three and the rest from article 2.

That would do some of the work, then you'll have to read the folders from the disk to get just which files you want to start and end with - but it would narrow you to reading a maximum of 2 overlapping folders.

In memcached you could create an Array which indexes each artilce. Like:

$article_counts = array($article1_counts, $article2_counts, ...);

But add and remove each index depending on how many times its hit to keep it small. eg: if there isn't any hits on article 2 for an hour it definitely should not have lived in memcached for an hour since thats more than the cache lifetime on SQUID I bet...

I've never worked with memcached or Squid though so I'm just guessing here..

I'm choosing just the "count's" here cause it seems the easiest to update. Say a new comment is made, just have to update the count in the folder the comment file will be in...

Am I even close to what you're getting at?

Hi ether,

Your reply seems a nice pointer for me to think and work out. Aaah a way out i feel so. Thanks. I would give a try after some thoughts and post an update.

Thanks once again for the patience.

Harish

If you cache the number or files in each folder like: (using your example)

Article
{Folder 1}
Reply 1
Reply to 1 as 1.n
reply to 1.n as 1.n.n
......
{Folder 2}
Reply 2
{Folder 3}
Reply 3

Say you cache an array.

$article1_counts = Array('folder1'=>3, 'folder2'=>1, 'folder3'=>3);

say your limit is 5 (to make the example work).

Then 1 would be folder1 and folder2 and 1 file from folder 3.
Then 2 would be 2 files from folder three and the rest from article 2.

That would do some of the work, then you'll have to read the folders from the disk to get just which files you want to start and end with - but it would narrow you to reading a maximum of 2 overlapping folders.

In memcached you could create an Array which indexes each artilce. Like:

$article_counts = array($article1_counts, $article2_counts, ...);

But add and remove each index depending on how many times its hit to keep it small. eg: if there isn't any hits on article 2 for an hour it definitely should not have lived in memcached for an hour since thats more than the cache lifetime on SQUID I bet...

I've never worked with memcached or Squid though so I'm just guessing here..

I'm choosing just the "count's" here cause it seems the easiest to update. Say a new comment is made, just have to update the count in the folder the comment file will be in...

Am I even close to what you're getting at?