Network Backups poor throughput
#1
Scooby Regular
Thread Starter
Join Date: Nov 2001
Location: Leeds - It was 562.4bhp@28psi on Optimax, How much closer to 600 with race fuel and a bigger turbo?
Posts: 15,239
Likes: 0
Received 1 Like
on
1 Post
Does anybody use Veritas Backup exec 8.6 to do their backups, and backup to disk.. and then backup the disk to tape?
Story is:
I am experimenting in backing up some of our critical servers as they aren't finishing out of hours... at the moment backups strugle to finish... managing approx 25mb/min which is poor...
I have got two test servers set up at the moment, both on the same subnet, however I have added two additional network cards and put a crossover cable between the two so that I should see 100mbps per sec... but I am only managing 40mb per min in backup exec... (fair enough one server is only an old pc... Other Compaq Proliant)
What transfer rates are people seeing on their DLT Backup devices??
What speed what you expect to see over a 100mbp/s switched lan? 12.5MB per Sec??
My theory is setup a totally seperate switched network for all servers in addition to main network, on a diferent subnet... and then backup servers to a IDE raid array in the server (Size and Cost would prohibit SCSI) Then during the day back this up using ADIC DLT Autoloader (this should theoretically cope with 200gb) per device.
Hows everybody else do it? Comments Suggestions?
David
Story is:
I am experimenting in backing up some of our critical servers as they aren't finishing out of hours... at the moment backups strugle to finish... managing approx 25mb/min which is poor...
I have got two test servers set up at the moment, both on the same subnet, however I have added two additional network cards and put a crossover cable between the two so that I should see 100mbps per sec... but I am only managing 40mb per min in backup exec... (fair enough one server is only an old pc... Other Compaq Proliant)
What transfer rates are people seeing on their DLT Backup devices??
What speed what you expect to see over a 100mbp/s switched lan? 12.5MB per Sec??
My theory is setup a totally seperate switched network for all servers in addition to main network, on a diferent subnet... and then backup servers to a IDE raid array in the server (Size and Cost would prohibit SCSI) Then during the day back this up using ADIC DLT Autoloader (this should theoretically cope with 200gb) per device.
Hows everybody else do it? Comments Suggestions?
David
#2
Scooby Senior
Join Date: Oct 2000
Location: Zurich, Switzerland
Posts: 3,105
Likes: 0
Received 0 Likes
on
0 Posts
Where to start
First thing is your LAN for the current 3.3Mbps throughput - I'd check for errors on both the server NIC and the switch ports. Better to hard-code both ends to full duplex operation too.
3.3Mbps throughput is slow - but there's a million things that could be causing it (both on the servers & the LAN). Most likely is a LAN bottleneck somewhere - packet sniffing is the best way of confirming/denying this.
In your test environment, do you *know* the traffic is passing over the x-over cable connection?
Best is to start with basics - use two devices with only a x-over cable connecting them and running only tcp/ip ptotocol. Then use ftp (if NT, not using MS domain authentication, use local accounts!) to transfer a large file between the harddisk's. This forms your baseline and is the theoritcal maximum given your hardware/os combination.
You can then try the same thing using a transfer medium higher up the osi layer - you don't mention what OS you're running, but if it's NT try using NetBIOS next. If it's not then your backup application.
Assuming your LAN, and the device connections into it, is OK then start looking at the OS configuration. Too many protocols with an unoptimised binding order all the way through to a badly fragged swap space could be contributing.
With MS OS's (and a good hardware platform) I doubt you'll ever see more than 60Mbps throughput anyway - even with a x-over cable. 12-20Mbps seems the norm for a "slapped-together" vanilla box.
Richard
First thing is your LAN for the current 3.3Mbps throughput - I'd check for errors on both the server NIC and the switch ports. Better to hard-code both ends to full duplex operation too.
3.3Mbps throughput is slow - but there's a million things that could be causing it (both on the servers & the LAN). Most likely is a LAN bottleneck somewhere - packet sniffing is the best way of confirming/denying this.
In your test environment, do you *know* the traffic is passing over the x-over cable connection?
Best is to start with basics - use two devices with only a x-over cable connecting them and running only tcp/ip ptotocol. Then use ftp (if NT, not using MS domain authentication, use local accounts!) to transfer a large file between the harddisk's. This forms your baseline and is the theoritcal maximum given your hardware/os combination.
You can then try the same thing using a transfer medium higher up the osi layer - you don't mention what OS you're running, but if it's NT try using NetBIOS next. If it's not then your backup application.
Assuming your LAN, and the device connections into it, is OK then start looking at the OS configuration. Too many protocols with an unoptimised binding order all the way through to a badly fragged swap space could be contributing.
With MS OS's (and a good hardware platform) I doubt you'll ever see more than 60Mbps throughput anyway - even with a x-over cable. 12-20Mbps seems the norm for a "slapped-together" vanilla box.
Richard
#3
Scooby Regular
Thread Starter
Join Date: Nov 2001
Location: Leeds - It was 562.4bhp@28psi on Optimax, How much closer to 600 with race fuel and a bigger turbo?
Posts: 15,239
Likes: 0
Received 1 Like
on
1 Post
I have monitored the cards, and I am getting no bad packets.. it is not going via a switch so no obvious latency there.
I am fairly sure it's not network bottleneck.
I know that the traffic is coming over the other link, as main network is 10.0.0.0 and test network is 11.0.0.0 and I am going to server by IP address not NetBios Name. I have only bound tcp/ip to the relevant adaptors with no ipx etc... I am thinking about playing with optional tcp/ip parameters next.. Will try the ftp transfer... Would have to install IIS though.
I am fairly sure it's not network bottleneck.
I know that the traffic is coming over the other link, as main network is 10.0.0.0 and test network is 11.0.0.0 and I am going to server by IP address not NetBios Name. I have only bound tcp/ip to the relevant adaptors with no ipx etc... I am thinking about playing with optional tcp/ip parameters next.. Will try the ftp transfer... Would have to install IIS though.
#4
Scooby Regular
Thread Starter
Join Date: Nov 2001
Location: Leeds - It was 562.4bhp@28psi on Optimax, How much closer to 600 with race fuel and a bigger turbo?
Posts: 15,239
Likes: 0
Received 1 Like
on
1 Post
oops... didnt mean to post that just yet...
I think the other problem could be the other test box (desktop pc) could be the FSB speed < 100mhz should therefore limit the top speed of the card rendering it incapable of doing 100mbp/s also the ide controller is on the same bus and not capable of something like ata100. I will be happy with 40mbps
Oh yeah... OS in NT4 Server Sp6a
David
[Edited by David_Wallis - 10/18/2001 10:25:06 AM]
I think the other problem could be the other test box (desktop pc) could be the FSB speed < 100mhz should therefore limit the top speed of the card rendering it incapable of doing 100mbp/s also the ide controller is on the same bus and not capable of something like ata100. I will be happy with 40mbps
Oh yeah... OS in NT4 Server Sp6a
David
[Edited by David_Wallis - 10/18/2001 10:25:06 AM]
#5
Scooby Regular
Join Date: Mar 1999
Location: Essex
Posts: 1,681
Likes: 0
Received 0 Likes
on
0 Posts
We backup servers onto a 320Gb snap server via a dedicated cisco switch. Recently had a trouble with 2 servers going real slow.
Only after installing updated drivers did I see that the NICs in the servers running half duplex, whereas the switch is forced to 100Mb/full. I forced the servers to 100Mb/full and bingo - big speeds.
12.5Mb/sec - thats the theoretical limit based on pure transfer. Add overheads of packet headers, the fact that the backup software must create the backup file on the fly etc..most I've seen is about 10.5-11.
So..get the latest drivers and force everthing onto 100mb/full.
Only after installing updated drivers did I see that the NICs in the servers running half duplex, whereas the switch is forced to 100Mb/full. I forced the servers to 100Mb/full and bingo - big speeds.
12.5Mb/sec - thats the theoretical limit based on pure transfer. Add overheads of packet headers, the fact that the backup software must create the backup file on the fly etc..most I've seen is about 10.5-11.
So..get the latest drivers and force everthing onto 100mb/full.
#6
Scooby Regular
Thread Starter
Join Date: Nov 2001
Location: Leeds - It was 562.4bhp@28psi on Optimax, How much closer to 600 with race fuel and a bigger turbo?
Posts: 15,239
Likes: 0
Received 1 Like
on
1 Post
According to our cisco / network guys.. I would be better off using full duplex (100% efficiency) as no collisions are possible..
Either way the Native transfer rate of the backup device is 90mb/min
Even though I want to backup to disk... I am only getting 25MB /per min....
and we say that I should see 12.5mb per second..
Something wrong somewhere...
David
Either way the Native transfer rate of the backup device is 90mb/min
Even though I want to backup to disk... I am only getting 25MB /per min....
and we say that I should see 12.5mb per second..
Something wrong somewhere...
David
#7
Scooby Regular
iTrader: (7)
Join Date: Jun 2001
Location: Stalking Kate Beckinsale
Posts: 4,265
Likes: 0
Received 0 Likes
on
0 Posts
On good clean LAN's, I tend to get 10-10.5MB's per sec sustained transfer rate. That's non duplex mostly.
No comparison, as the tape drive is bottlenecking the performance, but My DDS4's in the office tend to back up at around 180-190MB/Min across the LAN/Locally.
Cheers,
Nick
No comparison, as the tape drive is bottlenecking the performance, but My DDS4's in the office tend to back up at around 180-190MB/Min across the LAN/Locally.
Cheers,
Nick
Trending Topics
#8
If you are using NT 4 be very careful about assuming full duplex will provide better throughput. It *should* but there was a bug in NT which meant full-duplex screwed the network stack. Whilst working for NatWest (Big NT Users!) we came across that many many times whilst testing NIC and Network hardware compatability.
Off the top of my head I cant remember which service pack allegedly fixed it (if at all). I'll try to find the knowledge base article.
Do not leave any NICS auto-sensing, try setting both to full-duplex and then both to half-duplex and compare file transfer times.
Some Intel NICs have adapative technolgy which dynamically adjust the time between frames. This can work well in a shared envrionment but can slow a direct connection.
Off the top of my head I cant remember which service pack allegedly fixed it (if at all). I'll try to find the knowledge base article.
Do not leave any NICS auto-sensing, try setting both to full-duplex and then both to half-duplex and compare file transfer times.
Some Intel NICs have adapative technolgy which dynamically adjust the time between frames. This can work well in a shared envrionment but can slow a direct connection.
#9
Scooby Regular
Thread Starter
Join Date: Nov 2001
Location: Leeds - It was 562.4bhp@28psi on Optimax, How much closer to 600 with race fuel and a bigger turbo?
Posts: 15,239
Likes: 0
Received 1 Like
on
1 Post
By changing from an onboard card to a pci card I have managed to get the data transfer rate up to 170mb /min. (this is 100mb Full Duplex) and I now reckon the bus speed is the bottle neck in the system as the processor on the desktop pc is nothing special. <200mhz. FSB is probably 66mhz... so limiting netcard.
I now want to try proliant to proliant.... going to rebuild 3000 with dual 450's after lunch so will post results.
David
I now want to try proliant to proliant.... going to rebuild 3000 with dual 450's after lunch so will post results.
David
#10
Scooby Regular
I'd switch off any AV software that may be running.... Did that here and noticed a big improvement in performance (but that was just local backups, not network backup).
#11
Scooby Regular
Thread Starter
Join Date: Nov 2001
Location: Leeds - It was 562.4bhp@28psi on Optimax, How much closer to 600 with race fuel and a bigger turbo?
Posts: 15,239
Likes: 0
Received 1 Like
on
1 Post
I would unfortunatley I am also the antivirus administrator for our organisation, we are currently using netshield on nt servers and if you load performance montitor there are some counters that you can add, and you can do files scan / sec etc and it does'nt seem to be causing any problems.. besides I would rather have a virus free backup...
David
David
#12
Scooby Regular
Thread Starter
Join Date: Nov 2001
Location: Leeds - It was 562.4bhp@28psi on Optimax, How much closer to 600 with race fuel and a bigger turbo?
Posts: 15,239
Likes: 0
Received 1 Like
on
1 Post
Well I would post the results... but sh1tty netware servers that crash at 17:10 on a friday when you want to go home...
30mb free on data, tried to down the server and it hung dismounting the volume... how did I know that was going to happen...
worked all day yesterday fixing it... (I do nt not netware... bad sign all ready..)
brought it up with server -nas, edited startup.ncf and added set automatically repair bad volumes = off... then ran vrepair only 17 errors... just got a txt message saying its gone down again... oh dear never got that message... can feel another netware server being migrated off digital hardware and on to a compaq with NT.
only 45gb of data... best get my backup solution sorted quick coz these get backed up by adsm on aix rs6000's
David
30mb free on data, tried to down the server and it hung dismounting the volume... how did I know that was going to happen...
worked all day yesterday fixing it... (I do nt not netware... bad sign all ready..)
brought it up with server -nas, edited startup.ncf and added set automatically repair bad volumes = off... then ran vrepair only 17 errors... just got a txt message saying its gone down again... oh dear never got that message... can feel another netware server being migrated off digital hardware and on to a compaq with NT.
only 45gb of data... best get my backup solution sorted quick coz these get backed up by adsm on aix rs6000's
David
#13
Scooby Regular
iTrader: (5)
Join Date: Mar 2001
Location: Cheshire
Posts: 2,895
Likes: 0
Received 0 Likes
on
0 Posts
I currently run for my customer,v7.3 of Backup Exec and within that there's something called "Agent Accelerator".I assume that this is still available in newer releases of the software as I notice you use 8.6?
Agent Accelerator allows servers to locally process data and send it in one continuous stream rather than fits and starts.....
I seem to remember when I first implemented v7.3 and didn't use the accelerator data throughputs were pretty poor on our fully switched 10/100 network (20mb or so per second) but after investigation I read about the accelerator and thus installed it onto remote servers.Things have improved significantly! (Upto 90mb per second)
Hope this gives you another angle to look at?
Nick..
Agent Accelerator allows servers to locally process data and send it in one continuous stream rather than fits and starts.....
I seem to remember when I first implemented v7.3 and didn't use the accelerator data throughputs were pretty poor on our fully switched 10/100 network (20mb or so per second) but after investigation I read about the accelerator and thus installed it onto remote servers.Things have improved significantly! (Upto 90mb per second)
Hope this gives you another angle to look at?
Nick..
#15
Scooby Regular
Thread Starter
Join Date: Nov 2001
Location: Leeds - It was 562.4bhp@28psi on Optimax, How much closer to 600 with race fuel and a bigger turbo?
Posts: 15,239
Likes: 0
Received 1 Like
on
1 Post
As far as I know... we have to use the agent accelerator.. with 8.6. It is installed however. Like I said I think the limiting factor at the moment is the bus speed.. time permitting I will be looking at this next week.
David
David
#17
Whilst it wouldn't slow it down so much, is the data you're backing up compressed? If so you'd be better turning off hardware compression on the DLT - certainly turn off software compression if it's enabled (unlikely) and your machines are relatively low spec.
Even 66mhz FSB should easily manage 100Mbs - unless the box was doing any other heavy I/O at the time?
Dean
Even 66mhz FSB should easily manage 100Mbs - unless the box was doing any other heavy I/O at the time?
Dean
#18
Scooby Regular
Thread Starter
Join Date: Nov 2001
Location: Leeds - It was 562.4bhp@28psi on Optimax, How much closer to 600 with race fuel and a bigger turbo?
Posts: 15,239
Likes: 0
Received 1 Like
on
1 Post
Licencing isnt a problem... we are well covered on that..
Compression shouldnt matter as I am trying to backup over the network to disk so that the backups will finish during the night, and then get backed up on to dlt during the day...
The test machine was only a 133 with IDE disk though...
Will try compaq 3000 to 3000 tomorrow.
David
Compression shouldnt matter as I am trying to backup over the network to disk so that the backups will finish during the night, and then get backed up on to dlt during the day...
The test machine was only a 133 with IDE disk though...
Will try compaq 3000 to 3000 tomorrow.
David
#19
Scooby Regular
Thread Starter
Join Date: Nov 2001
Location: Leeds - It was 562.4bhp@28psi on Optimax, How much closer to 600 with race fuel and a bigger turbo?
Posts: 15,239
Likes: 0
Received 1 Like
on
1 Post
Used different servers and transfer rate went up to 263.9mb/min (full duplex 100mb connection)
Realised that remote agent acellerator wasnt installed and I got 9mb a min when installed... yes 9mb.... removed and transfer rate shot up.. Rung support tried loads of things.. in the end we changed the connection to 100mb half duplex and we are seeing 660mb throughput now...
Now building another test server... to perform some proper testing...
David
Realised that remote agent acellerator wasnt installed and I got 9mb a min when installed... yes 9mb.... removed and transfer rate shot up.. Rung support tried loads of things.. in the end we changed the connection to 100mb half duplex and we are seeing 660mb throughput now...
Now building another test server... to perform some proper testing...
David
Thread
Thread Starter
Forum
Replies
Last Post