Topic: Search Engine Announcement

Message:

Posted by: Gene (Feb 21, 2006 5:16pm)

[ANNOUNCEMENT]

As of yesterday at 5PM (Eastern US), the Café's search engine was shut down due to being the cause of the recent slowdowns. The specific cause is keyword searches. All other functions of the search engine do not appear to cause any problems. Links to the search engine have been removed, except the "view posts in the last 30 days" link on the member profile pages. I will be removing that soon so there is no confusion.

I will be looking at everything as soon as I can. There will be a lot of thought put into this, so it will not be a quick process. My goal is to come up with a solution that does not change any of the search engine's features, yet still provides accurate results within an acceptable period of time.

[SEARCH ENGINE HISTORY]

For the benefit of those who have not been with us since the beginning, the search engine has been a focal point of many discussions and has undergone several revisions. The original version was very slow and extremely limited, but it did offer one thing that nearly all present-day forums don't: Exact phrase searches. This was possible because it used a very slow database function. It didn't take long for that to start crashing the Café, so we had to come up with a solution. I don't think anybody ever really realized it, but version 2 was an almost complete copy of the search engine used with phpBB2. That one, too, started to crash the forums. I thought it would last a long time, but in reality, it didn't last that long at all. There were still complaints about it, too, most of which were because it didn't offer exact phrase searches. I made some changes to the original phpBB design to include certain limits so that it wouldn't crash the Café. It worked - but there was a deluge of complaints. People were very vocal about it. Everybody seemed to want the whole smash: Two-character words, being able to search all posts without regard to when posted and, of course, exact phrase searches. It took me a while, but I came up with a version I really thought would do it. It answered (nearly) everyone's needs and wants, but at a delayed price. That one was shut down yesterday. Soon I will begin considering options and thinking about the next version: V4. Who knows where that one will end up?

[GENERAL THOUGHTS]

Trust me, this is not going to be an easy task. I fully understand how important the search engine is for the Café - indeed, for any site - and I will do my best, but please allow me the time I need to work it out. Technically speaking, there is nothing wrong with the search engine we have been using, it's just not efficient enough to handle the massive amount of data in use here. In fact, I am using the same search engine structure for two other sites with perfect success. It's simply a matter of data. For those who have any interest in the figures, the total size of the Café database is over 1.4GB. (That's 1.4 gigabytes, folks. :eek: ) The search engine itself currently contains 282,387 keywords, with a total keyword/post-ID data size of nearly 235MB. These are gargantuan numbers to deal with on a single server. Is there an answer? I don't know that yet. I'm only one guy. Other forum packages have several developers, or a complete team. However, if there *is* a solution, I will find it. Hey, I figured out how to offer exact phrase searches in a modern forum, didn't I? :bg:

And now, let the process begin ... hang in there, everybody!

Message:

Posted by: pegasus (Feb 22, 2006 3:21am)

Apparently you can use google to search specific websites, including the Café, but I cannot remember how exactly. If anyone knows, then could they please post it here for others?

Message:

Posted by: pegasus (Feb 22, 2006 3:26am)

As a quick fix then you can use Google as an internal site search engine.

Example= torn and restored site:www.themagiccafe.com

Type in what you are searching for, then leave a space, then type the following. site:wwwthemagiccafe.com

Message:

Posted by: GarySumpter (Feb 22, 2006 6:31am)

The only problem is that the google results wont be up to date, but better than nothing I suppose.

Can you not just move th ecafe to a more poweful server?

Gary

Message:

Posted by: Edith (Feb 22, 2006 7:21am)

I am looking forward to see the result... of all your good and hard work, Gene!

Edith

Message:

Posted by: pegasus (Feb 22, 2006 8:37am)

The following link allows google to operate as an internal search engine. Perhaps it can be used until a proper fix has been found.

http://www.javascriptkit.com/script/script2/google_site_search.shtml

Message:

Posted by: Marco S. (Feb 22, 2006 11:22am)

Hope you will find a solution quickly.

Message:

Posted by: evolve629 (Feb 22, 2006 12:07pm)

Perhaps you can install an auto delete function on the members search so that key words are not memorized by the search engine to prevent overloading the system. Just a thought!

Message:

Posted by: Tom Jorgenson (Feb 22, 2006 2:03pm)

We all appreciate your past efforts and grand successes....we'll survive this one....good luck. Does Starbucks deliver???

Message:

Posted by: mshanker (Feb 22, 2006 3:47pm)

Gene, several sites--including U.S. government ones--embed the Google search engine into their sites. Couldn't that be a solution to the problem?

I for one think it's a shame to disable the search feature.

Manoj

Message:

Posted by: pegasus (Feb 22, 2006 4:14pm)

Quote:

On 2006-02-22 08:37, pegasus wrote:
The following link allows google to operate as an internal search engine. Perhaps it can be used until a proper fix has been found.

http://www.javascriptkit.com/script/script2/google_site_search.shtml

Already mentioned manoj

Message:

Posted by: Gene (Feb 22, 2006 4:22pm)

Yes, I'll be doing something about that when I can get to it.

Message:

Posted by: Bill Hallahan (Feb 22, 2006 6:32pm)

Someone mentioned searching using http://www.google.com/ in another topic. Just add "site:themagiccafe.com" to your search string without the double quotes.

You can even search titles in a specific forum using Google. For example, to search the Pick a card...any card forum titles for an exact match to the string "three card monte" you would enter the following into the Google search text box.

Quote:

"three card monte" "Pick a card...any card » »" site:themagiccafe.com

The double quotes cause a case insensitive exact string match. You could leave them off on the first quoted string and you'd find topics that contain "three", "card", and "monte", but not necessarily together or even in that order.

Of course, someone could put "Pick a card...any card » »" in a post too and it would show up in the title search I describe, but I doubt anyone has posted that, and I hope they won't.

This is not quite as good as the Magic Café's title search was since each forum's titles have to be searched separately, and also the search data is not updated in real time. But it does work pretty well.

Message:

Posted by: Foucault (Feb 23, 2006 10:37am)

Quote:

On 2006-02-22 18:32, Bill Hallahan wrote:
but I doubt anyone has posted that, and I hope they won't.

erm..you just did! ;)

Message:

Posted by: MikeRaffone (Feb 23, 2006 1:07pm)

I can't find the link "view posts in the last 30 days". Has this feature been removed too?
I am trying find a post from a few days ago and can't find it.

Message:

Posted by: Gene (Feb 23, 2006 1:43pm)

Yes. Read my first post.

Message:

Posted by: stannmaple (Feb 23, 2006 9:00pm)

Gene, Do you really live in Rockford, MI? That's crazy! You're like my next door neighbor. I live in Grandville. Why did I never notice this before? Crazy!
Dann

Message:

Posted by: Gene (Feb 23, 2006 10:25pm)

Actually, you just reminded me that I have to change that. I live in Howard City now. If you ever want to stop by and learn how it all started just let me know. :)

Message:

Posted by: willmorton (Feb 24, 2006 7:19am)

Gagging for the search engine to return! Google is ok, but the Café-specific one is vital!

Message:

Posted by: ClouDsss (Feb 24, 2006 9:09pm)

I really miss the search function :(

all the best and hopefully a solution is found :)

Message:

Posted by: Jordini (Feb 24, 2006 9:10pm)

Let the reposting of topics formerly discussed, BEGIN!

Message:

Posted by: equivoque (Feb 24, 2006 9:28pm)

People are just going to post the same topics over and over now...

Message:

Posted by: Alym Amlani (Feb 25, 2006 4:07am)

Quote:

On 2006-02-24 21:28, equivoque wrote:
People are just going to post the same topics over and over now...

Sigh, they already do.

Message:

Posted by: Steve Brooks (Feb 25, 2006 4:55am)

Sorry folks for the big inconvenience, but that's just the way it is - at least for now. Are folks going to post a topic that has already been discussed? You bet they are, but even when the search engine was functioning properly many of our members never bothered to use it. Such is life though and we'll have to live with that for a bit while our resident genius Gene figures out a new design. Thanks one and all for your patience and understanding in this important matter. :smoke:

Message:

Posted by: fccfp (Feb 25, 2006 6:53am)

Gene & Steve,
Please don't let the complainers get you down. The Café has been a wonderful place for me to discuss magic w/ others. I have "met" folks that I would not have possibly run into any other way. I have learned a lot, shared some of the little that I know, & kidded around with a lot of really terrific magicians willing to share what they know and offer advice. Some of the recommendations given in the forums have helped me tremendously in improving my skills and structuring my act. I have gotten many ideas that would have never occurred to me otherwise.

The Café is THE place to go for help when you are having difficulty with a effect. Mechanical, pacing, patter, routining, you will find it all here. Just recently, someone gave me a page reference in one of the classic magic resource books. Just doing that gave me what I needed to know. Just having that page number saved me untold hours fiddling with the prop just to figure out how to set it up. I still can't believe how much is available here.

With all the Café has given us, at no cost, how can anybody complain about a technical problem now and then? Anyone that uses computers learns that they often do not work as anticipated. (My next novel will be "When Programs Collide”.) :bg:

Thank you to everyone that works on making the café a truly magical place. We do appreciate it.
:applause: :applause: :applause: :applause: :applause:

Sincerely,
Bruce J. Langrock, CFP
Magician

P.S. Illegitimi Non Carborundum

Message:

Posted by: The Great Dave (Feb 25, 2006 7:42am)

I will offer you $10 per year as a volunteer subscription to your service. I pay more than that for my golf magazine and enjoy your site more. Perhaps others may join in, and with the extra revenue additional expansions can be made to the features of your site, such as the search engine technology. And Gene won't work for free. You may offer me a contribution option anytime. (Ladies and Gentlemen, if you enjoyed the show please Pass the Hat among yourselves!)

Best Wishes,
Dave

Message:

Posted by: Mind_Magic (Feb 25, 2006 9:10am)

Meantime you can seach as following:

At Yahoo Web Page, you select "Advanced" search Option located at the top right side.

Check the "Only search in this domain/site" option, and in the box you type: www.themagiccafe.com

Then into "any of the words" box you write what you are looking for.
Or for more specific search use "all of this words" box.

Finally click "Yahoo Search" option at right corner of the screen.

Guest help you

Message:

Posted by: Steve Hook (Feb 25, 2006 11:35am)

Ditto:

"Thank you to everyone who works on making the café a truly magical place. We do appreciate it."

And, "Patience is a virtue."

And and, "Virtue is an example or kind of moral excellence."

Message:

Posted by: todsky (Feb 25, 2006 12:03pm)

Good luck, Gene. Take your time, do it right. Your efforts here are appreciated.

Message:

Posted by: Gene (Feb 25, 2006 2:55pm)

Thanks guys. And don't worry, I'm not "down" about anything, it's just something else to deal with, that's all. I'm not going to lose any sleep over it.

Many people here do not know who I am, or even what I contribute here, because I don't post about magic in the various rooms. There's a reason for that: I don't *know* anything about magic! I've read many times that the Café is important to people because of its wealth of information and being able to contact people that would otherwise be difficult to do, not to mention many more reasons that I can't think of at the moment. At the same time, the Café is important to me too, but for different reasons.

I've been here from the beginning and have watched the Café blossom into what it is. Most of the credit belongs to Steve because of his vision and his ability to bring his dream into reality. For me, while I can claim part of the credit, more important for me is how much the Café has taught me as a programmer. I wouldn't be able to do many of the things I can do today if it weren't for the Café. Some of it still surprises me. (Where else could I find such a proving-ground as the Café?) The search engine is one of those things, even though it is currently offline. I know the codes are pretty solid, it's just a fact that the mass of data here is too much for a single-processor server to handle. Unfortunately, that's where we're at.

The Great Dave was wrong in his assumption that I won't work for free. The fact is, I do. Right here. This is, and will always remain, a free site, and as such, I don't make any money from it. What I pay I write off on my hosting business taxes. :) It's because of that that I feel I'm just as much a contributor to this site as anybody else.

Message:

Posted by: leefoley3 (Feb 25, 2006 9:50pm)

Gene, you rock, brother!

Got no doubt that you'll get 'er taken care
of and the new version will be worth the wait.
I think part of the problem for us is that we
have been spoiled by what The Café offers us.
A lot of us are used to being able to hop on
here and find exactly what we're looking for
fairly quickly and go from there.
As mentioned by others above, we greatly appreciate
what you contribute to The Café!!

Any thing worth having... :bwink:

take care, bud.

Lee

Message:

Posted by: kerpa (Feb 26, 2006 12:59am)

Is there a need to add a second server? If so, then perhaps voluntary contributions as mentioned above should be considered. Please let us know! I second all the grateful things said about this great site!
kerpa a/k/a Michael Miller
Chicago area

Message:

Posted by: revlovejoy (Feb 26, 2006 2:12am)

I'd pay a monthly fee for this site. It has been worth more to me than any other website, ever. You could certainly take donations to cover server costs. Totally anonymously, or just designate users as "sponsors" if recognition is desired to be given. Still keep it a free site, but don't be shy that contributions would help.

Message:

Posted by: Gene (Feb 26, 2006 2:49pm)

I appreciate the thought, and I'm sure Steve does too, but he is adamant that the Café remains a free service, and I agree with him. To answer kerpa, yes, we need a more powerful server to handle the load. Donations are all fine and well, the problem is that donations can't be relied upon. Somebody has to have the money to pay for the server if donations aren't there, and frankly, nobody directly connected to the Café has that kind of money. Don't worry, though. We'll figure out something. We always do! :)

Message:

Posted by: The Great Dave (Feb 26, 2006 3:30pm)

How much cash do you need for the second server?

Best Wishes,
Dave

Message:

Posted by: Gene (Feb 26, 2006 4:07pm)

Replacement server, actually. What I'm looking at is $400/month, a little more than double what I'm paying now. Ouch! :eek: With that much money I could buy my favorite car - new!

Message:

Posted by: GarySumpter (Feb 27, 2006 4:33pm)

There are a lot of paying advertisers on here. Where is that money going? I'm SURE that would easily cover any server costs?

Gary

Message:

Posted by: Gene (Feb 27, 2006 4:53pm)

Most advertisements here are for trade, such as ads in magazines and things like that. Those aren't actual money-makers, they're meant to help the Café grow even more.

Message:

Posted by: Steve Brooks (Feb 27, 2006 4:53pm)

Gary,
Appearances can be very deceiving. First, we will deal with the search engine, so that's not really an issue, just a small inconvenience for all of us.
Concerning the advertisers, many of the banners are trade outs or favors to our friends, so any thoughts that we are rolling in piles of cash need to be eliminated right now.

In the meantime, Gene has placed a temporary search solution for the Café and that will have to suffice until we decide the best way to approach the problem.
Rest assured we will find a solution - we always do. Problems like these are part of the Café becoming so large and are nothing more than growing pains. That said, I suggest everyone go back to enjoying the forums and know that we will find an appropriate solution. Thank you.