SDB

Hi All

We have had a small breakthrough.

We have managed to track the methods of the denial of service attacks we have had on the scoobynet over the last couple of months.

These have caused the site to be completely shut down due to a huge demand placed on the server by a (presumed) malicious source.

They have been attacking the "search" Function.

It turns out that the search function takes up and enormous amount of processor commitment and about 25MB of memory per search. This means that you only need to run about 20 searches to cause all the spare memory to be chomped up.

As far as we can see, this is the only process that requires this level of resources so we have decided to switch it off...

Do not fear though, we are in the process (anyway) of producing a ScoobyNet CD which will allow you to browse and search the archives locally.

We will just speed up the completion of this to try to get it done as soon as possible.

I hope this doesn't cause a problem...

thoughts?

Simon

chiark

I did wonder if it was this. There's been a couple of times when I've tried to search for something and the whole system has been running like a dog...

Maybe a database based BBS would be the best option?

So perhaps these weren't DDOS attacks at all? It's a viscious circle: no response from server, hence click again, hence more loading on server, hence click again...

If there's 129.227.x.x, it could be me. Oops.

ian/555

Simon that Explains a few probs I have been having lately.

Count me in for a CD, maybe make it upgradeable via a download?

Ian

Neil Smalley

That's a great shame, simon. The search feature was soo useful. BUT it's better it's shut off than the site keep crashing and slowing down.

Some people huh. The site is here to help people with all kind of cars and what do they do. F*cking wreck it.

Any chance of having a downloadable version to save on CD mastering?

MartinM

..but the search is dead useful!!! An offline version (only) would be a bit of a limitation IMHO (distribution, costs etc etc)

Are we really getting malicious DOS attacks, or is it just that we have a large concurrent user population that are occasionally causing 20 concurrent searches?

I appreciate the problem (didn't know UBB was so bad at using memory when searching though) but IIRC ('cos its gone already ) can't there be some halfway house like restricting the search to just some forums, and/or only back a month or so at a maximum - does it still use loads of memory when there is a date restriction of 'last 30 days'?? I expect so

..or stripping the UBB directories that hold the threads for each forum of all those threads older than (say) a month into online but unsearchable forums?

..or making a user put in their username and password before they can initiate a search ( would that stop external malicious DOS attacks?)..although I appreciate your reluctance to modify UBB code - but loads of people do it as you know

Martin

Boost

Yes, a shame - I have found the search engine very useful. Would it be possible to have it switched on "off-peak" which I am guessing is around 12am - 8am GMT for Scoobynet. I appreciate this may have the reverse effect of concentrating all the searching into one period but not too many users would be inconvenienced if it ran slowly. CD would work OK if updates could be downloaded I suppose.

[This message has been edited by Boost (edited 30 November 2000).]

harryh

Bugger, the search was great, I use it all the time. Oh, God, it's not me causing the DDOS attacks is it? Sorry

MartinM

A bit off topic, but I wouldn't jump for a database-based Scoobynet without some careful analysis. If the analysis has been done and justifies it then ignore the rest of this...

UBB is dead efficient when retrieving forum thread lists and the threads themselves, because they're all static HTML files, so loads of concurrent users are (relatively) not a real problem. UBB takes a hit when you
- add a reply, because it has to insert your reply into the existing static file for the thread which causes a lot of string manipulation
- do a search, 'cos it has to do lots of string searching.

Depending on the ratio of 'surfing' users to 'updating' users, UBB could be the best solution. Do we know how many concurrent users we have and the read/write hit rates? At a wild guess I'd say we could have had 50 reads per minute and 0.5 write per minute. Quite a big difference

A DBMS based Scoobynet would take a big hit just retrieving forum thread lists and the threads themselves, because each will cause some SQL (presumably) to run for every browsing users - and unless its doing some clever caching of the front page of General Questions and every thread on it (but, hey that's what UBB effectively does ) then your TPS requirement could be v.large...

..and it could end up going like a 3-legged dog in treacle ... or needing some v.expensive tin to run it on.

Simon...I'm happy to help thinking about this, but a voice call would be much more efficient. Let me know if you want a chat

Martin

SDB

Hi All

OK...

This was going to be a secret until it was released, but I guess this turn of events has made it appropriate to announce it now..

As one of the many ways we are earning money to help pay for scoobynet we have been developing a scoobynet CD app...

The idea is that we will include the technical archives (and maybe others depending on what we can fit on / what the demand is) and also a quarterly CD magazine including video clips from track days / test sessions at bruntingthorpe, etc, etc (you get the picture).

There will be articles including a "Lift-Off-Oversteer Workshop" including video footage. There will be product reviews, group tests, including a "Scooby vs Challenger" section where I will test a challenging marque or model against the different models of impreza (In a genuinely un-biased way (so don't expect - "scooby wins again" on every article! )

This was going to happen anyway...

The fact that the search facility has had to be switched off meant that I felt it appropriate to mention the CD archives.

I would appreciate support on this for the following reasons...

1) I cannot begin to convey the amount of time that goes into running scoobynet. From me, my staff and all of the moderators to whom I am eternally grateful. So putting in extra facilities (like modding the UBB software) and then dealing with the inevitable problems that causes, is a time commitment I hope can be avoided.

2) The the load on the server has increased by such an enormous amount in the last few months (which is great!), that we are going to have to upgrade the server 6 months ahead of schedule. This means there is a large additional cost that we were not expecting.

I think it will be clear to everyone that my FIRST priority is to continue to support and improve ScoobyNet for all of us, and I can assure you that we are working hard to make it happen.

There have been complaints over the last few weeks of speed issues, and we are taking steps to sort that out (new hardware, etc) but switching off the search facility is going to make quite a difference immediately so the main part of scoobynet will run much better.

When we upgrade / re-write the BBS software we will probably be able to switch the search facility back on, but until then we the best solution is to switch it off.

It IS a shame, but not the end of the world. We should be proud of having a community based site that has problems like this due such popularity and rapid growth.

All the best

Simon

PS. Re downloading the archives. You would probably be surprised to know that they are around 2.5GB!!!!

joni

Must admit, would miss the search function. Has been invaluable for a relative virgin like myself. I was so looking forward to making one of those patronising posts to an obvious newbie question.... "This has been covered on the board soooo many times. Do a search on blah blah blah" !! But that is a point?

Gary Foster

A lot of RDBMS's allow you to - ehum - 'cache' commonly used queries. Not a good term but you get the idea.

Eg a view may be "join posts, users order by posts.date" (errm) would be pre-calculated view which is always stored in memory, so the result is already known when the query is submitted.

There is then obviously some (clever ?) code which updates the result of this if any affected fields change.

Anyway blah blah blah, RDBMS's are made for this kinda thing, there's years of development in them and they are the dogs danglies

Gary

Wino Wilky

Darn shame search rocked.

So how much is this application CD going to cost?

I'll buy it just for the content you have mentioned.

Gary Foster

Simon

Yippee ! That sounds great, excellent excellent ideas, all of them.

Any chance of putting a 'sound selection' on there, eg for

1) This is what piston slap sounds like
2) This is what a Scoobysport back box sounds like
3) This is what det sounds like

Bugger, just ignore me I'm sure your far too busy to pander to my every whim

Just tell me when it's available and how much and you can have my credit card details

Gary

Neil Smalley

2.5Gb! I was expecting 200-300Mb! I suspect if we trimmed out all the 'Muppet' stuff over the years you'd be able to send the other useful stuff out on a floppy disk

So what special features will this CD have? Directors commentry or scoobynet best of deleted threads.

How is this going to be issued 2.5Gb is about 4 CD's. Are you going to do
"Now that's what I call scoobynet 1"
or "The Best scooby BBS in the world ever Part 2"?

Or simply and maybe more sensibly by date?

For those with a DVD ROM this would fit on a single DVD, if SDB had a writer.

I agree, lemmie know how much and i'll be there!

[This message has been edited by Neil Smalley (edited 30 November 2000).]

merlin

The search facility was a very useful feature and helped me resolve many questions without having to resort to posting. For newbies, being able to search the archive helps stop the same question being asked repeatedly. The newbies who most need the search feature are the group who are least likely to fork out for a CD. The FAQs document should help answer the common questions, but in my experience, having comments from people with first hand experience is better than edited FAQs.

Working with relational databases systems in the day job, the first thing you tell the client is to have enough physical memory to avoid disk swapping as this is so expensive in time in comparison to memory access. Hence they tend to over specify on the memory side, particularly as memory is cheap when compared to the cost of the software & our time. As a quick solution, can memory be thrown at this problem to increase the number of possible parallel searches beyond the maximum anticipated load? If users are also educated to restrict their searches to off peak hours, then maybe this could workaround the issue until a system with a larger capacity can be implemented.

The alternative CD solution introduces admin costs for scoobynet of producing and distributing the CDs. Also, the users like myself, would need to keep getting an upto date CD to have the latest information. This reminds me of the way Microsoft keeps money rolling in by 'encouraging' users to update to the latest & greatest version of their software. Sorry if that's a bit of an extreme analogy

SDB having just read your post about the CD - sounds like a great idea - but keep the search online too.

Going down the CD route, then I can foresee people posting requests on the general forum for those who have the latest CD, to do a search for them. This is the sort of thing that happens on the genealogy lists where only a few people have access to the 19th century census records on CD.

If the search feature has finally met its end, then what about putting the archives in files on a ftp server so they can be downloaded and searched locally? It would be interesting to know the size of such a download. Using zipped files, the file sizes shouldn't be too large as the data is text and not binary. The data could of course be spread across many files, one per forum and in the case of the general forum, several files.

SDB, I've just read your comment that the archive is 2.5GB Is that zipped up or normal text & is there any binary in there? If there is binary, then that could be kept out of the files that you download.

BTW, what happened to the search facility that allowed you to search across all open forums? This disappeared a few weeks ago & was really useful

[This message has been edited by merlin (edited 30 November 2000).]

[This message has been edited by merlin (edited 30 November 2000).]

Dave T-S

Well i'm back - firewalled, megavirus protection and raring to go

SDB/Joni
Hee Hee - it's wonderful news that the archives are disabled. Now when someone comes on and asks "can I blah blah" I can say "search the archives" and Muppet them when they come back and say what archives

robski

Simon,

have you considered that now maybe the time to charge a small subscription?

Send the CDs to everyone who subscribes, similar to SIDC where you can view the online stuff, but you get more in the post.

To me its more of a community on scoobynet than SIDC, its just that SIDC often gets the credit.

Maybe some of the better things can be limited to scoobynet members, rather than non members. Loads of issues Im sure, but just to promote some thought.

Another consideration could be to add some "shares". If scoobynet really really takes off then maybe one day it will be worth some serious cash. Some of the other web sites do this to get people interested. Its in the members benefit then to ensure its better than ever.

robski