Author Topic: Diner Down 6/30/13  (Read 1247 times)

Reverse Engineer

  • Administrator
  • Newbie
  • *****
  • Posts: 28
    • View Profile
Re: Diner Down 6/30/13
« Reply #15 on: July 02, 2013, 03:05:14 am »
Latest from Tech Support, and my response.  IMHO, this looks like a DNS ATTACK.

Quote from: Tech Support
   
Hello,

Thank you for contacting Support.

This is for the ticket # 11433840. I've taken all the database configuration information for the Simple Machines Forums for doomsteaddiner.org including the database password and the Forum is now displaying that it is in Maintenance mode. A possible cause for the sites on different accounts going down could be that the database queries is being hit. There is a limit of 75000 queries per hour and once that has been reached, the connection to the database will cease. The other accounts mentioned don't indicate constant high amounts of queries but spikes could cause the database to hit the limit.
If you have any further issues, please reply to this message and we will be happy to assist you.

Alan G.
Technical specialist

Quote from: RE
   
If we are getting 75K queries per hour, then it must be some kind of DNS attack, IMHO. There is no way we have that many legitimate Human Eyeballs hitting our site that fast, and less likely Peter's sites could be hit on by so many people.

Some sort of filter needs to be in place to prevent so many queries. I am willing to shut all my boards down to Members Only if absolutely necessary, but to do that I need to have the Admin Page of SMF back up for me to make those changes. I would prefer not to do that, and just filter out BOT queries and so forth, but if it is the only means available, I am open to this.

I now am pretty certain we are at WAR with the NSA.

RE

Surly1

  • Newbie
  • *
  • Posts: 15
    • View Profile
Re: Diner Down 6/30/13
« Reply #16 on: July 02, 2013, 11:31:57 am »
As much as I am fond of the tinfoil beanie, I am not yet ready to believe we are making enough noise to attract the NSA's attention. Apparently there are increasingly sophisticated ways to mount these attacks and amplify them.

found this--
http://www.infoworld.com/t/internet/possibly-related-ddos-attacks-cause-dns-hosting-outages-220050

It would be interesting to know if any other servers of the hosting company have been hit.

agelbert

  • Newbie
  • *
  • Posts: 4
    • View Profile
Re: Diner Down 6/30/13
« Reply #17 on: July 02, 2013, 01:06:34 pm »
Getting so many queries is not logical.

You may think me paranoid but it occurred to me that there is one person that VERY unhappy with the doomstead diner that may just be on a revenge rampage inventing all sorts of spurious accusations based on imagined "security" threats.

Why? Hell hath no fury and all that.

I suspect K has a hand in this.  >:(

haniel

  • Newbie
  • *
  • Posts: 5
    • View Profile
Re: Diner Down 6/30/13
« Reply #18 on: July 02, 2013, 01:10:43 pm »
I think this is a wordpress issue. 

I'm seeing the majority of the traffic to the site being hits to the web page http://doomsteaddiner.org/blog/wp-admin/admin-ajax.php which looks to be the admin front end.

The 10 simultaneous connection limit applies to all the databases on the diner, not just the forum.  Peter or RE may be hosting other forums there as well. 

We're getting a lot of bot traffic - this is Bing hits over the last day:

1046 Total from: 157.55.35.112
1116 Total from: 157.55.32.111
1116 Total from: 157.56.92.144
1119 Total from: 157.56.93.61 

With great content comes great traffic - at least Bing wants to index our work.

Also we're seeing a lot of "Human" traffic:

4914 Total hits from a cox communications subscriber, probably in a hole he dug in Virginia
4681 Total hits from a Choopa.com customer in New Jersey
2934 Total from a user near Rosemount (Check your email, you know who you are)

Two freaking days to get the info to be able to diagnose properly.

Not sure of a solution yet.  Yves at Naked Capitalism had WP issues for a long time, and that site is not as technology heavy as here.



Surly1

  • Newbie
  • *
  • Posts: 15
    • View Profile
Re: Diner Down 6/30/13
« Reply #19 on: July 02, 2013, 01:25:04 pm »
Quote
4914 Total hits from a cox communications subscriber, probably in a hole he dug in Virginia

That's a bunker, BTW... ;D

haniel

  • Newbie
  • *
  • Posts: 5
    • View Profile
Re: Diner Down 6/30/13
« Reply #20 on: July 02, 2013, 01:27:39 pm »
Anyone who is logged into the wordpress admin screen in the Diner, please log out until further notice.

It would appear that each contributor may be keeping one db connection open each if they are not logged out.   

I'll take the SMF forum out of maintenance mode after lunch to see if the reduced connections allow the forum DB to be available.



Surly1

  • Newbie
  • *
  • Posts: 15
    • View Profile
Re: Diner Down 6/30/13
« Reply #21 on: July 02, 2013, 01:34:42 pm »
As I told haniel in a FB chat, I have one wp/admin connection logged in at home. Am at work now. Will log that one off as soon as I hit the threshold, probably about 6PM EDT.

RE, are you getting anything at all from your "hired guns?"

haniel

  • Newbie
  • *
  • Posts: 5
    • View Profile
Re: Diner Down 6/30/13
« Reply #22 on: July 02, 2013, 01:35:23 pm »
Take a look at the bandwidth growth of the Diner over the last three months.

Month   Unique visitors   Number of visits     Pages              Hits   Bandwidth
Apr 2013       21448                  48985                 404276   1312099   23.39 GB

May 2013       28462                  65669                 474226   2021750   51.31 GB

Jun 2013       25082                  60683                 539307   2630317   72.37 GB

Bandwidth has grown significantly faster than hits, which indicates more data per visitor.  The multimedia is bringing in traffic - a lot of traffic.



agelbert

  • Newbie
  • *
  • Posts: 4
    • View Profile
Re: Diner Down 6/30/13
« Reply #23 on: July 02, 2013, 01:37:41 pm »
Monsta told me once he was able to log into the admin accidentally. I have never been able to do so.

Haniel,
You might send Monsta an e-mail. Maybe the fact that he is in England is multiplying the hit count somehow.

agelbert

  • Newbie
  • *
  • Posts: 4
    • View Profile
Re: Diner Down 6/30/13
« Reply #24 on: July 02, 2013, 01:41:35 pm »
Haniel,
Does that mean we will have to post less video so the bandwidth remains manageable?

I'll stop posting videos if that helps stabilize the site.

haniel

  • Newbie
  • *
  • Posts: 5
    • View Profile
Re: Diner Down 6/30/13
« Reply #25 on: July 02, 2013, 01:45:46 pm »
No, the bandwidth should be manageable, we have to figure out the excessive connections issue. 

The site -> forum should only take up on connection.  Problem is, none are available when the site tries. 

I'll talk with my hosting experts.



Reverse Engineer

  • Administrator
  • Newbie
  • *****
  • Posts: 28
    • View Profile
Re: Diner Down 6/30/13
« Reply #26 on: July 02, 2013, 02:40:01 pm »
There are only 5 people with Admin priviledges on WP, me, Peter, Surly, WHD & Monsta.  I closed all my open window before I left the cabin, but since my connection is through my cell phone, once I leave that computer is not connected to the net anymore.

The multimedia is not hosted on our server at all, we just link to it.  All the Podcasts are on Soundcloud, all the videos are on You Tube.  This should not affect our bandwidth usage at all, other than the fact these things draw more visitors and page hits.

I haven't checked yet but nothing yesterday from the Paid Support help, just from the regular support team.  Haniel has also dropped in on the ticket, pointing out that our DB doesn't show anything like 75,000 queries/hour.

Far as being on the same server as Peter, possible but not likely.  I bought my account long after Peter bought his, I just handed him the password so he could do the setup.  Besides that, after the last outage, I had to upgrade to a server with fewer websites on it, I don't think Peter has had to do that with the Ocean Falls sites.  I don't have any other sites active.

RE

Reverse Engineer

  • Administrator
  • Newbie
  • *****
  • Posts: 28
    • View Profile
Re: Diner Down 6/30/13
« Reply #27 on: July 02, 2013, 02:41:49 pm »
Quote
4914 Total hits from a cox communications subscriber, probably in a hole he dug in Virginia

That's a bunker, BTW... ;D

Langley.  CIA or NSA for sure.

RE

haniel

  • Newbie
  • *
  • Posts: 5
    • View Profile
Re: Diner Down 6/30/13
« Reply #28 on: July 02, 2013, 03:49:39 pm »
Traffic increased from 25GB to 75GB in two months.  Impressive but it may have the hosting service rubbing their hands together in glee.

I don't see any problems beyond a highly restrictive number of connections to the database, imposed by the hosting service.

They recommended:

The 75,000 query limit is per database log in. A quick way to see if a database log in is suspended is to try to access phpMyadmin. If you get prompted for a password, create another user and try to log in to the same database. If it works, then the user is probably suspended.

So I create another user (more than once) and change the forum -> DB connection string to use the "new user" - and even that one cannot connect to the database.

The fact that repair_settings.php brings back details of smiley and avatar locations means that the "settings_repair.php" page is able to connect to the DB - it tells you when it cannot and does not give details of file locations - the pointers are stored internally in the DB and I can see those pointers. 

So the hosting service are suspending a new user connection with only one connection against that user.  Either they are counting all user against all databases towards a maximum of "10" or there is somethign untoward going on with the hosting service.

What could that be?
It is recommended that our mutual client optimize his database or upgrade from shared to a VPS if he has outgrown shared hosting.

The databases were optimized as soon as I got in and backed them up.  So this appears to be a push for more money.

I've also tried to connect to a copy on an external server, but they appear to have port 3306 blocked.

one of the two AC's have failed in our server room and the temperature is climbing, so if there's no update for a few hours, we're scrambling to make sure our servers stay cool in the 109* heat.





Reverse Engineer

  • Administrator
  • Newbie
  • *****
  • Posts: 28
    • View Profile
Re: Diner Down 6/30/13
« Reply #29 on: July 02, 2013, 05:38:46 pm »
Traffic increased from 25GB to 75GB in two months.  Impressive but it may have the hosting service rubbing their hands together in glee.

I don't see any problems beyond a highly restrictive number of connections to the database, imposed by the hosting service.

They recommended:

The 75,000 query limit is per database log in. A quick way to see if a database log in is suspended is to try to access phpMyadmin. If you get prompted for a password, create another user and try to log in to the same database. If it works, then the user is probably suspended.

So I create another user (more than once) and change the forum -> DB connection string to use the "new user" - and even that one cannot connect to the database.

The fact that repair_settings.php brings back details of smiley and avatar locations means that the "settings_repair.php" page is able to connect to the DB - it tells you when it cannot and does not give details of file locations - the pointers are stored internally in the DB and I can see those pointers. 

So the hosting service are suspending a new user connection with only one connection against that user.  Either they are counting all user against all databases towards a maximum of "10" or there is somethign untoward going on with the hosting service.

What could that be?
It is recommended that our mutual client optimize his database or upgrade from shared to a VPS if he has outgrown shared hosting.

The databases were optimized as soon as I got in and backed them up.  So this appears to be a push for more money.

I've also tried to connect to a copy on an external server, but they appear to have port 3306 blocked.

one of the two AC's have failed in our server room and the temperature is climbing, so if there's no update for a few hours, we're scrambling to make sure our servers stay cool in the 109* heat.

Have you dropped any of this info on the Ticket?

Also for everyone else...GOOD NEWZ! Haniel has booted the DB on his server in a Temp location and it WORKS!

The Diner will be BACK.

RE