|
 |
| | | Author: | Timothy_Smith  | | Posted: | Dec 11, 2010 06:25 | | Subject: | Stop purging data | | Viewed: | 401 times | | Topic: | Suggestions | | Status: | Discarded | |
|
| My suggestion: stop purging data.
It's the 21st century, mass storage is cheap.
There's no at all reason to purge data ever.
|
|
| | | | | |
| | | | | Author: | legomadsteve | | Posted: | Dec 11, 2010 06:30 | | Subject: | Re: Stop purging data | | Viewed: | 118 times | | Topic: | Suggestions | |
|
| In Suggestions, Timothy_Smith writes:
| | My suggestion: stop purging data.
|
You should purge your Avatar, it's terrible!
|
|
| | | | | | | | | |
| | | | | | | Author: | redbeardlegoman | | Posted: | Dec 11, 2010 06:46 | | Subject: | Re: Stop purging data | | Viewed: | 96 times | | Topic: | Suggestions | |
|
| In Suggestions, legomadsteve writes:
| | In Suggestions, Timothy_Smith writes:
| | My suggestion: stop purging data.
|
You should purge your Avatar, it's terrible!
|
Flame on!
|
|
| | | | | | | | | |
| | | | | | | Author: | JoeMomma | | Posted: | Dec 11, 2010 10:26 | | Subject: | Re: Stop purging data | | Viewed: | 78 times | | Topic: | Suggestions | |
|
| In Suggestions, legomadsteve writes:
| | In Suggestions, Timothy_Smith writes:
| | My suggestion: stop purging data.
|
You should purge your Avatar, it's terrible!
|
At least his is original! Your Avatar is generic maybe you should take some of
your time to replace it instead of insulting people on here!
JoeMomma
|
|
| | | | | |
| | | | | Author: | bb166186 | | Posted: | Dec 11, 2010 06:47 | | Subject: | Re: Stop purging data | | Viewed: | 103 times | | Topic: | Suggestions | |
|
| In Suggestions, Timothy_Smith writes:
| | My suggestion: stop purging data.
It's the 21st century, mass storage is cheap.
There's no at all reason to purge data ever.
|
Data purging is not done to save storage cost. It is done to cut data access
time. My guess is all bricklink visitors are served by only 1 processor.
Boris.
|
|
| | | | | | | | | |
| | | | | | | Author: | Timothy_Smith | | Posted: | Dec 11, 2010 06:59 | | Subject: | Re: Stop purging data | | Viewed: | 99 times | | Topic: | Suggestions | |
|
| In Suggestions, B0RIS writes:
| | Data purging is not done to save storage cost. It is done to cut data access
time.
|
Minor technical issue which can be addressed.
|
|
| | | | | | | | | |
| | | | | | | Author: | Locutis | | Posted: | Dec 11, 2010 10:23 | | Subject: | Re: Stop purging data | | Viewed: | 67 times | | Topic: | Suggestions | |
|
| In Suggestions, B0RIS writes:
| | In Suggestions, Timothy_Smith writes:
| | My suggestion: stop purging data.
It's the 21st century, mass storage is cheap.
There's no at all reason to purge data ever.
|
Data purging is not done to save storage cost. It is done to cut data access
time. My guess is all bricklink visitors are served by only 1 processor.
Boris.
|
I don't know how Bricklink works, but I know on my work server, I can search
through AND display over 20,000 records in less than 1 second. We have one single
server, with 2x 500GB hard drives on RAID 1 (which means they duplicate the data
over 2 drives), running Apache2, with a 2.8 GHz Pentium processor.
If I want to search the "archived data" which takes 10,000 records and magically
combines it into 1 for speedier access (I programmed it to archive records over
2 years old 10,000 at a time into one database entry), there's 200,000 records
to search, it takes only several seconds to search, compile, and display the
information.
Again, I don't know how Bricklink works, but I'm far from a professional programmer,
and I made the site at work operate efficiently this way. We never purge ANY
data, and have access to all data going back to 1997 when we started computerizing.
I'm one single computer person at work, and computer work isn't even my job,
I do it in my "spare time" while trying to manage the company.
Cameron
|
|
|
| | | | | | | | | | | | | |
| | | | | | | | | Author: | AggieSava | | Posted: | Dec 11, 2010 11:04 | | Subject: | Re: Stop purging data | | Viewed: | 69 times | | Topic: | Suggestions | |
|
| In Suggestions, locutis writes:
| | In Suggestions, B0RIS writes:
| | In Suggestions, Timothy_Smith writes:
| | My suggestion: stop purging data.
It's the 21st century, mass storage is cheap.
There's no at all reason to purge data ever.
|
Data purging is not done to save storage cost. It is done to cut data access
time. My guess is all bricklink visitors are served by only 1 processor.
Boris.
|
I don't know how Bricklink works, but I know on my work server, I can search
through AND display over 20,000 records in less than 1 second. We have one single
server, with 2x 500GB hard drives on RAID 1 (which means they duplicate the data
over 2 drives), running Apache2, with a 2.8 GHz Pentium processor.
If I want to search the "archived data" which takes 10,000 records and magically
combines it into 1 for speedier access (I programmed it to archive records over
2 years old 10,000 at a time into one database entry), there's 200,000 records
to search, it takes only several seconds to search, compile, and display the
information.
Again, I don't know how Bricklink works, but I'm far from a professional programmer,
and I made the site at work operate efficiently this way. We never purge ANY
data, and have access to all data going back to 1997 when we started computerizing.
I'm one single computer person at work, and computer work isn't even my job,
I do it in my "spare time" while trying to manage the company.
Cameron
|
But how many people are using your work server? There are 152,351 registered
members of Bricklink as I'm writing this, and Bricklink already has an extensive
database to go through as it is for every single person. I can imagine how if,
as suggested, BL has only one processor, a database containing all the orders
and all the forum posts since the beginning would slow considerably for all these
users.
But as also already suggested, hardware is getting cheaper and cheaper.
--Tony
|
|
|
| | | | | | | | | | | | | | | | | |
| | | | | | | | | | | Author: | Locutis | | Posted: | Dec 11, 2010 11:38 | | Subject: | Re: Stop purging data | | Viewed: | 64 times | | Topic: | Suggestions | |
|
| Our internal work server (used for invoicing, accounting, etc.) has only 5 users.
However, our external webserver (online store, auction site, gold/silver bullion
quotes, etc.) receives many millions of hits per month. We send over 60Gb of
html data over our fibre internet connection every month.
Our customer database consists of 80,000 + active customers.
On our external server (same hardware as the internal, 2.8 GHz processor, 2x500
GB RAID 1 hard drives) we operate a standard webserver, an auction website (with
a current catalog of 4,000 active items, and 40,000 archived items) which accepts
and processes bids in realtime, an online store with over 2500 active items,
and a gold and silver bullion website which provides realtime quotes to 10's
of thousands of customers every minute.
Our website is high profile as well. I am one person programming in "spare time",
and have made all of this work, and work efficiently.
Cameron
In Suggestions, AggieSava writes:
| | In Suggestions, locutis writes:
| | In Suggestions, B0RIS writes:
| | In Suggestions, Timothy_Smith writes:
| | My suggestion: stop purging data.
It's the 21st century, mass storage is cheap.
There's no at all reason to purge data ever.
|
Data purging is not done to save storage cost. It is done to cut data access
time. My guess is all bricklink visitors are served by only 1 processor.
Boris.
|
I don't know how Bricklink works, but I know on my work server, I can search
through AND display over 20,000 records in less than 1 second. We have one single
server, with 2x 500GB hard drives on RAID 1 (which means they duplicate the data
over 2 drives), running Apache2, with a 2.8 GHz Pentium processor.
If I want to search the "archived data" which takes 10,000 records and magically
combines it into 1 for speedier access (I programmed it to archive records over
2 years old 10,000 at a time into one database entry), there's 200,000 records
to search, it takes only several seconds to search, compile, and display the
information.
Again, I don't know how Bricklink works, but I'm far from a professional programmer,
and I made the site at work operate efficiently this way. We never purge ANY
data, and have access to all data going back to 1997 when we started computerizing.
I'm one single computer person at work, and computer work isn't even my job,
I do it in my "spare time" while trying to manage the company.
Cameron
|
But how many people are using your work server? There are 152,351 registered
members of Bricklink as I'm writing this, and Bricklink already has an extensive
database to go through as it is for every single person. I can imagine how if,
as suggested, BL has only one processor, a database containing all the orders
and all the forum posts since the beginning would slow considerably for all these
users.
But as also already suggested, hardware is getting cheaper and cheaper.
--Tony
|
|
|
|
| | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | Author: | Locutis | | Posted: | Dec 11, 2010 12:27 | | Subject: | Re: Stop purging data | | Viewed: | 78 times | | Topic: | Suggestions | |
|
| I forgot to mention something. Last year I looked forward towards the future,
and investigated High Availability Linux for our servers. It is actually quite
easy to setup and guides are everywhere for this. What does this do for you?
Well, using standard off-the-shelf hardware, and open source software, you can
build a cluster of Linux web and database servers for less than $5k that offer
you the ability to run and maintain a website that can server over 100,000 concurrent
database and web connections/second. As well, it provides data backup and redundancy
over 2+ computers. If one fails, using something called heartbeat, another server
simply takes the load while you replace the failed unit. Hard drives can be
swapped in and out and be added to the pool of data storage easily. Built in
to this system is load balancing. Just google "how to build a high availability
linux cluster" and you will find lots of instructions for this. I purchased
4 identical server hardware machines in 2009 with the hope of eventually putting
them into place. Currently we are using 2 of the 4 units, with the other 2 units
fully assembled and configured to be ready for an inplace replacement in case
of a catastrophic failure. What did all of this cost (including 15x 500 GB hard
drives for instant replacement of failed drives)? Around $4k.
I don't know if Bricklink already employs this, but at work I was looking into
it and never found the time to implement the cluster system. A cluster is easy
to maintain once setup. If you find your site is getting busier and load on
the server is overwhelming, simply setup another cluster machine, bring it online,
and add it to the pool. If you had 2 machines, and you add 1, now you've increased
your available processing capacity by 50% and done nothing to the existing 2
machines. As far as users are concerned, if you add or take away any machines
from the cluster, it only affects the time to access the site. Need to perform
maintenance and upgrade the OS or other software? Take everything but one offline
(which means the site still works), upgrade and test the "spares", then switch
everything over to the new system, and bring the older one up to date by adding
it now as a "spare" to the new system.
The clustering mechanism in Linux can scale easily from 2 to unlimited.
The software is called "drdb" and "heartbeat". When employed with RAID, and
an interconnected gigabit connection between the computers, you have a very fast
responding system, which scales easily with load over time, and highly reliable
because of the fault tolerance inherent in having more than 1 identical webserver
online. You can even have your servers setup this way in different locations,
different cities, even different countries! All of the data is mirrored and
kept in sync across all machines, and is online and accessible all of the time.
I'm just saying it's possible, can be done by virtually anyone in the computer
field who is knowledgeable on website setup with the guides that are out there,
and it's very inexpensive.
In Suggestions, locutis writes:
| | Our internal work server (used for invoicing, accounting, etc.) has only 5 users.
However, our external webserver (online store, auction site, gold/silver bullion
quotes, etc.) receives many millions of hits per month. We send over 60Gb of
html data over our fibre internet connection every month.
Our customer database consists of 80,000 + active customers.
On our external server (same hardware as the internal, 2.8 GHz processor, 2x500
GB RAID 1 hard drives) we operate a standard webserver, an auction website (with
a current catalog of 4,000 active items, and 40,000 archived items) which accepts
and processes bids in realtime, an online store with over 2500 active items,
and a gold and silver bullion website which provides realtime quotes to 10's
of thousands of customers every minute.
Our website is high profile as well. I am one person programming in "spare time",
and have made all of this work, and work efficiently.
Cameron
In Suggestions, AggieSava writes:
| | In Suggestions, locutis writes:
| | In Suggestions, B0RIS writes:
| | In Suggestions, Timothy_Smith writes:
| | My suggestion: stop purging data.
It's the 21st century, mass storage is cheap.
There's no at all reason to purge data ever.
|
Data purging is not done to save storage cost. It is done to cut data access
time. My guess is all bricklink visitors are served by only 1 processor.
Boris.
|
I don't know how Bricklink works, but I know on my work server, I can search
through AND display over 20,000 records in less than 1 second. We have one single
server, with 2x 500GB hard drives on RAID 1 (which means they duplicate the data
over 2 drives), running Apache2, with a 2.8 GHz Pentium processor.
If I want to search the "archived data" which takes 10,000 records and magically
combines it into 1 for speedier access (I programmed it to archive records over
2 years old 10,000 at a time into one database entry), there's 200,000 records
to search, it takes only several seconds to search, compile, and display the
information.
Again, I don't know how Bricklink works, but I'm far from a professional programmer,
and I made the site at work operate efficiently this way. We never purge ANY
data, and have access to all data going back to 1997 when we started computerizing.
I'm one single computer person at work, and computer work isn't even my job,
I do it in my "spare time" while trying to manage the company.
Cameron
|
But how many people are using your work server? There are 152,351 registered
members of Bricklink as I'm writing this, and Bricklink already has an extensive
database to go through as it is for every single person. I can imagine how if,
as suggested, BL has only one processor, a database containing all the orders
and all the forum posts since the beginning would slow considerably for all these
users.
But as also already suggested, hardware is getting cheaper and cheaper.
--Tony
|
|
|
|
|
| | | | | |
| | | | | Author: | Reki_Lobsheek | | Posted: | Dec 11, 2010 11:28 | | Subject: | Re: Stop purging data | | Viewed: | 50 times | | Topic: | Suggestions | |
|
| as nobody else seems to ask this, I will: to what purpose?
I mean, I wouldn't mind if promise could be made it wouldn't slow down the website,
but I'm just wondering why it would be necessary?
Reki
|
|
| | | | | | | | | |
| | | | | | | Author: | Locutis | | Posted: | Dec 11, 2010 11:34 | | Subject: | Re: Stop purging data | | Viewed: | 54 times | | Topic: | Suggestions | |
|
| In Suggestions, Reki_Lobsheek writes:
| | as nobody else seems to ask this, I will: to what purpose?
I mean, I wouldn't mind if promise could be made it wouldn't slow down the website,
but I'm just wondering why it would be necessary?
Reki
|
Instead of deleting the data, they could archive it like I have, and move it
to another database. Set a script to run and archive it to an alternate database.
If you want access to the archived information, have it come from the archived
database, which gets a lower priority on the processor then current data.
Why keep data, why not? The storage of a database for the data is nearly free
(500 GB hard drives are $40 currently, and can hold decades of database text
data).
|
|
| | | | | | | | | | | | | |
| | | | | | | | | Author: | matthewcrandall | | Posted: | Dec 11, 2010 12:09 | | Subject: | Re: Stop purging data | | Viewed: | 53 times | | Topic: | Suggestions | |
|
| In Suggestions, locutis writes:
| | In Suggestions, Reki_Lobsheek writes:
| | as nobody else seems to ask this, I will: to what purpose?
I mean, I wouldn't mind if promise could be made it wouldn't slow down the website,
but I'm just wondering why it would be necessary?
Reki
|
Instead of deleting the data, they could archive it like I have, and move it
to another database. Set a script to run and archive it to an alternate database.
If you want access to the archived information, have it come from the archived
database, which gets a lower priority on the processor then current data.
Why keep data, why not? The storage of a database for the data is nearly free
(500 GB hard drives are $40 currently, and can hold decades of database text
data).
|
Okay, I'll play along...
Why keep some of this archived at all? What purpose does it serve?
I'm not against archiving data; but let's look at what gets archived first. Just
about everything here is considered data.
Off-topic stuff, bad jokes, "cancelled" replies to topics, and "Hi I'm a new
person" entries...is there really a need for that after say, nine months to a
year?
Catalog changes and additions...sure. Those need to be kept. But a lot of that
gets looked at, so it should remain on the main server. That's a lot of entries
right there. I'm not sure how it works, but store inventories probably exist
somewhere, too.
Then there's the questions that come up: how often do you move stuff to the archives?
Annually? Quarterly? And, there has to be a way to access and search those archived
items easily. That takes a little time; each item has to have a separate entry
to be totally searchable. Yes, I know you can set a script to run the transfer.
But there has to be a script ready to go, or it has to be written.
Yes, I understand memory is cheap these days. But it still has to be purchased,
installed, configured, and those items archived moved and made accessible...and
all without disrupting service so that everyone doesn't have a fit. (I would
say this time of year would be a bad time for the service to be down for more
than an hour, for example.)
I'd rather have faster processing server(s), and some very selective pruning
of data. Where I sit, that pays better dividends in the long run.
Play Well and Prosper,
Doc Crandall
"The Brick Detective"
|
|
|
| | | | | | | | | |
| | | | | | | Author: | Brickwilbo | | Posted: | Dec 11, 2010 13:06 | | Subject: | Re: Stop purging data | | Viewed: | 34 times | | Topic: | Suggestions | |
|
| In Suggestions, Reki_Lobsheek writes:
| | as nobody else seems to ask this, I will: to what purpose?
I mean, I wouldn't mind if promise could be made it wouldn't slow down the website,
but I'm just wondering why it would be necessary?
Reki
|
Agreed, voted the same as the last similair suggestion. 
|
|
| | | | | |
| | | | | Author: | Rbobo | | Posted: | Dec 11, 2010 13:24 | | Subject: | Re: Stop purging data | | Viewed: | 45 times | | Topic: | Suggestions | |
|
| In Suggestions, Timothy_Smith writes:
| | My suggestion: stop purging data.
It's the 21st century, mass storage is cheap.
There's no at all reason to purge data ever.
|
At very least, if we could have data available for 13 months, that would help
greatly come tax time.
|
|
| | | | | | | | | |
| | | | | | | Author: | wahiggin | | Posted: | Dec 11, 2010 13:29 | | Subject: | Re: Stop purging data | | Viewed: | 53 times | | Topic: | Suggestions | |
|
| In Suggestions, Rbobo writes:
| | In Suggestions, Timothy_Smith writes:
| | My suggestion: stop purging data.
It's the 21st century, mass storage is cheap.
There's no at all reason to purge data ever.
|
At very least, if we could have data available for 13 months, that would help
greatly come tax time.
|
I agree, 13-15 months would be great.
Wesley
|
|
| | | | | |
| | | | | Author: | eileenkeeney | | Posted: | Dec 12, 2010 02:48 | | Subject: | Re: Stop purging data | | Viewed: | 43 times | | Topic: | Suggestions | |
|
| In Suggestions, Timothy_Smith writes:
| | My suggestion: stop purging data.
It's the 21st century, mass storage is cheap.
There's no at all reason to purge data ever.
|
I strongly disagree.
I am guessing that you do not have any (or much) experience with the real world
of data, databases, data storage, etc ...
|
|
| | | | | |
| | | | | Author: | tomte | | Posted: | Jan 30, 2011 05:32 | | Subject: | Re: Stop purging data | | Viewed: | 50 times | | Topic: | Suggestions | |
|
| In Suggestions, Timothy_Smith writes:
| | My suggestion: stop purging data.
It's the 21st century, mass storage is cheap.
There's no at all reason to purge data ever.
|
No actually on the forum we could keep all messages. but I want orders that are
cancelled actually to purge entirely forever!
|
|
| | | | | |
| | | | | Author: | BLUSER_228233 | | Posted: | Jan 30, 2011 07:12 | | Subject: | Re: Stop purging data | | Viewed: | 63 times | | Topic: | Suggestions | |
|
| Agree. At least something as important as orders should not be purged as fast
as now. Should be kept for a minimum of 12 months. A modern dual-core based linux
server can handle a gazillion of database queries with no problems, and as for
storage, it's really dirt cheap now.
Nick
In Suggestions, Timothy_Smith writes:
| | My suggestion: stop purging data.
It's the 21st century, mass storage is cheap.
There's no at all reason to purge data ever.
|
|
|
|
|
|
USA, Colorado
United Kingdom, England
Canada, Alberta
Netherlands, Noord-Brabant
Belgium, Brussels
Germany, Bayern
Greece, Attica