AlexM

Simon,

Is it possible to change the search button default behaviour to search ONLY the forum you search from?

I'm sure most search requests should be restricted to one forum rather than all of them.

Cheers,

Alex

kryten

The problem is (I assume), that it is a text based search and the only way to perform it is with brute force!

Having been in the position before to try to have to rescue a (massive) HPUX box from the crippling effects of text based searching, I can appreciate the problems it causes when large searches are done.

Restricting the search by forum, or date range would help, but the problem still is a lot of people searching a lot of text == server slowdown.

I think the search on the CD is a great idea (probably more sales if its included with scoobynews), maybe then re-enable the search for 'new' stuff, until the next CD is released ......

gregh

Simon,

Problem is with CDs is that they are out of date so quickly!

What's the issue? Could you limit the amount of searches running at any one time, would that help?

How about off-loading the search to another server?

cheers,

greg

gregh

Any chance of search coming back at all? How about doing batch submission and results emailed back or something, or only allowing search in quiet times?

Seriously thinking of getting some big brake upgrades, and all the info is buried in here somewhere!

cheers,

Greg

Karlos

I agree, a search facility is needed. Being relatively new to the BB I find it a little embarassing having to ask the same question that's been asked months before.....

Karlos

Triggaaar

I agree with having it on during quiet periods (even just Midnight 'till 7am or something).

Greg - since you're thinking of upgrading brakes, you might like to know that I eventually got new pads from scoobymania (took just over a month) - I haven't posted any details yet, as I'm stressed out with bodywork repairs at the moment.

SecretAgentMan

<B>AGREED</B>

A manual search in the forums is a fuggin pain in the a$$, tried to find the (most excellent) link to Mr. Collinridge's PIAA webpage. Almost gave up...

Please Simon, bring back search to this excellent forum.

Regards

/Jerry - Sweden

[This message has been edited by SecretAgentMan (edited 02 January 2001).]

SDB

I know!

I'm sorry, but the server just couldn't take it!

I've thought about all sorts of alternatives, like the "only in quiet times",etc.. but this would just see people all attacking the search facility at the same time, which would almost definitely cause a problem.

We will upgrade the server soon which may make it possible to switch it back on, but it will not be cheap!

Another alternative is the ScoobyNews CD. Hald of it is a magazine, and the other half is the technical and general forum archives. The search function brings back the results in a split making it a far more rewarding experience anyway!!

This should be available soon.

Cheers

Simon

chiark

Jerry,

Rob's site:

SDB

OK, here's an idea...

The ScoobyNews CD is going to be £15 including p&p...

Would it be a good idea to release the Archive CD on it's own for about £10 or something?

This could be done almost immediately?

Something I forgot to say is... by buying the CD magazine, you are also supporting scoobynet, and helping us raise funds to buy the new server.

Cheers

Simon

PeteC

Simon,

The obvious problem with providing searchable archives on CD is that this will be out of date as soon as you create it - unless you bring back the search option for the threads that were not included on the CD.

Why has the use of search become such a problem, as I don't recollect it being an issue with Scoobynet 1?

Pete

SDB

Hi Pete

We could potentially release an archive CD every month. This way it is always fairly up-to-date. It is mainly historical information we all search for anyway.

In terms of the rason why it is now a problem...

ScoobyNet is hugely more popular and busy than it used to be. The demand on the server is far greater. We are running it on exactly the same spec server as it was running on before, and it ran perfectly up until about 2 months ago when it started to slow down.

Alex.. Your suggestion was indeed the first thing we did to try to solve the problem, but it didn't...

How it works...
As soon as you run a search, the search function spawns an instance of perl.exe which takes up 35MB!!! (I have no idea why it is so huge - maybe it is the sheer size of scoobynet (2.5GB) or maybe it is just bad coding?)

This 35MB object lasts for the duration of the search.

We were regularly seeing 10 searches going on at all times. The problem came when that went up to about 20 + ... that's 700MB of RAM just for searching. Add to that, the general browsing of scoobynet (between 100 and 250 people online all the time!) and posting, etc. Then the Chat server, etc.. it all adds up.

About 2 months ago everything started slowing down. It got to a stage where we needed to do something about it.

The problem was that the server kept running out of memory. It didn't matter how much more memory we stuck in it, it would still run suddenly. It seemed that one of the problems was, that people were clicking "search", it was taking a while, so they would click it again, and again (each time starting a new 35MB perl.exe session). This resulted in the server falling over and nobody could view the whole of scoobynet.

Considering that The rest of scoobynet uses around 100MB of RAM, we felt that the search (taking up 700MB (at good times!) was the thing to go).

You will notice that scoobynet has not gone down since we turned the search function off. It has been a welcome relief.

thoughts?

AWD

Is there a way of limiting the number of concurrent searches at any given time?

AWD

By that I mean:

Limit searches to e.g. 10 searches. That way the total memory taken up by searches is up to 350MB.

This could be lowered to 5 if 350MB is to high.

ric

PLEASE BRING THIS FUNCTION BACK

ric ;(

SDB

Hi AWD

There may be a way, and this is again something we considered before switching it off, but it means two things...

1) Development Time. We are all out of time at the moment with everthing else we are doing, so it's a case of priority.

2) It means changing the UBB code. This is something we took a decision to steer clear of as it could cause problems with the software, and give us a problem each time we want to upgrade to the latest version.

The best solution we came up with was...

Close it until we get scoobynet on the new server, then try it again. And in the mean time, make the scoobynet archives available on CD.

best regards

Simon

AWD

Simon

Fair enough.

Thanks for taking the time and effort to respond to our questions and suggestions.

Lee

removed my apparently terribly non helpful post.

[This message has been edited by Lee (edited 02 January 2001).]

Triggaaar

Simon,
With regards to too many people hitting it at one time if you enable the search facility for short periods: You could enable it at such a ridiculous hour, that people would only search for something that is really important to them (eg, they want to spend a lot of cash, so one day when they're alive at 3 in the morning, they can search). You could try different times, and find a balance. In the past, people have run searches *****-nilly, which wouldn't be the case if you chose appropriate hours.

Another crazy idea - if the search, more than anything else, is demanding a hardware upgrade, could you sell the search facility (instead of a soon out of date cd) such that users that have contributed to the new hardware get access to the facility?

Triggaaar

Another, more sensible idea.
Ban AWD from using scoobynet - 1797 posts in 5 months, and he's never even driven a subaru (his radio controlled version aside)

SDB

Hi Triggaaar

Yeah, sounds good, but... It would mean someone being up at those times to turn it on or off! ...

also to monitor it in-case of failure...

Lee

I have emailed you offline. Thank you for your offer of "help". Don't be baffled my friend...

With the clothing and CD Archive sales, and all of the other work we are putting into it, we will indeed be able to afford the server we had planned for 6 months time from now. This means that all will be well.

We may indeed go down the Unix route, etc..

The thing that baffles me is...

We have discussed this 1-1 many times. Why di you feel the need to mention all this here?

Best regards

Simon

MartinM

<anorak on><again><and with some slight simplifications>

How about....

...getting a third party "proper" search engine and pointing it at the forum directories. Perhaps like the Alta Vista search engine (available on

Gary Foster

Can't you just bung another 512Mb In the machine you've got ? that would only be a few hundred quid ? rather than 10K + on a new machine.

Admittedly not a long term solution, so here's my suggestion :-

Is there no way of suggesting changes to whoever develops UBB (such as the ability to limit the # of concurrent searches), this would fix the problem once and for all and would be included in all subsequent releases so little /no development effort required.

Other UBB users have I'm sure come across this problem already - what did they do ?

Gary

Ps liked triggaars idea of subscription based searching, but changing UBB code bad

Gary Foster

MartinM

Bit clunky as the webcrawlers are under the control of altavista, so there's no guarantee of how up to date the resultant links would be - ever clicked on a link to a page from a search engine and got nothing .....

Ps Google do a free one as well.

logiclee

I agree that no search facility is a real pain.

But...

Simon and others put a great deal of time and effort into Scoobynet so we can all enjoy it for FREE. I'm sure Simon is looking at all the options available, so lets all add our support and get some Scoobynet clothing ordered and standby for the Scoobynet CD.

Lee

SDB

Martin...

I actually think that is a really good idea (one I hadn't thought of), and that something similar to it may be a good temporary solution, but...

It will take time to set it up, and we hope to have the new server, etc up and running in the next few weeks anyway, so it may all work out best to just wait for that.

I promised I would constantly make scoobynet better, and it is the recent dramatic rise in popularity that has forced me to remove a benefit for the first time and I really hate it!

But leaving the search function in would have been far more negative than temporarily taking it away, so I promise, (whilst I am ALWAYS open to suggestions) I am working hard to get the search back as quickly as possible.

All the best

Simon

MartinM

Gary
The solution I posed gives you a search engine **OF YOUR OWN**. You actually run it on your own box. Its not related to the real Alta Vista web site at all..apart from it being the same "engine" of course. You control how often it crawls, so the latency of finding recent things is up to you

SDB
One more go then! What's happening to the current server when the new one comes? Could it become the Search engine server??
Or how about getting some IT company or ISP to host the search engine in return for some free banner ads or something? UBB is wonderful VFM as a BB, but we're talking about serious searching now, and even if the new box does UBB searches initially real quick, people will search more and more and it'll go slower and slower and slooower again and we'll have the same problem again. Getting a proper search engine on a separate box is the proper solution that I'd recommend to a client paying for my consultancy - but I realise, of course, that Scoobynet probably hasn't the financial reserves that my clients usually have

Iain Young

Hi Simon.

What operating system are you using for the BBS, (Unix or NT Server)? If you're using NT, (using IIS as the web server) then you could use the ISAPI DLL version of perl which should in theory reduce your memory footprint as well as speeding up the process(as only one copy is in memory, and all the search processes would use the same perl instance).

The main trouble with perl is that it is an interpreted language, and as such tends to be a tad inneficient when dealing with large amounts of data or bandwidth. A more efficient way would be to rewrite the search in C (or simlilar language), but I guess this may take some time....

Give me a mail if you want to discuss. I may be able to help....

Cheers,
Iain

MartinM

Sounds like we could use a GetTheSearchBackinScoobynetSoItDoesn'tAffectPerfor mance sub-committee that can be briefed by SDB on the minimum requirements and constraints and then reporting back?

There seems to be a few IT-literate people interested in helping out, so is there any chance of a 'virtual' subcommittee successfully coordinating some brainstorming and/or prototyping to see if we can do something about this - tactically and/or strategically

I can PROBABLY do some sort of demo/prototype using the AltaVista Search engine pointing at one or more forums and exposing it to the Internet

If there's a PERL expert, then they could get the eval version of UBB (fully functional, but with no search - but we maybe don't want that bit anyway!) and try some alternative approaches to realtime searching that are completely independent of UBB - thus not compromising the "we don't want to customise UBB at all" constraint at all

After all, a problem shared is everyone's problem , and as SDB seems superbusy, we might be able to do some parallel thinking

SDB - your call!...but there might just be some mileage in exploiting the 'comunity' spirit and expertise for mutual benefit

SDB

Hi All

OK, we have some developments...

As a spin off of Martin's idea I have been talking with someone about writing a bespoke search engine.

It will not be quite powerful as the UBB one in that you cannot say "search titles only", or whatever, but it should be pretty quick and take up next to no resources.

This is just about underway so I will keep you all informed.

Thank you to everyone who has contributed. It is very easy to jump in and be negative, but with the exception of a *very* small minority, everyone has been very positive in terms of finding a solution.

All the best, and thanks again.

Simon