PDA

View Full Version : Monthly traffic / data transfer / bandwidth accounting


PotentialClient
10-16-2004, 04:09 PM
Hi Dave & staff,

I would appreciate it if you could tell me:

1. Exactly which data counts towards the monthly data transfer on FreeBSD hosting plans with no dedicated IP?

2. Exactly which data counts towards the monthly data transfer on FreeBSD hosting plans with a dedicated IP?

3. How is data transfer measured and where can it be monitored by customers in real-time?

4. In case you do traffic accounting via SNMP, can a customer get read-only SNMP access to their traffic accounting data?

If so, do you have SNMP tools or the SNMP-Util Perl module installed on your systems so customers can run their own shell or perl warning / notification scripts as cron jobs?


Please feel free to give me technically detailed answers, I have a decade of network and Unix system administration experience.

FH-John
10-16-2004, 04:27 PM
1) E-mail, FTP, Web traffic all count towards your bandwidth usage.

2) Same as above

3) Not sure, Dave or someone else is better at answering that.

FH-Dave
10-16-2004, 07:31 PM
PotentialClient,


3. How is data transfer measured and where can it be monitored by customers in real-time?


Web, FTP, and Mail traffics are calculated daily by analyzing the log files. You can then see the traffic usage from your control panel.

Traffic is not calculated by SNMP. However, we do have SNMP installed on each web server (if not, you can request it to be installed) and you can then setup cronjob to utilize it.

Out of curiousity, have you taken a look at our Virtual Private Server (VPS) solution (fluidhosting.com/vps/)? VPS behaves as its own dedicated server with its own virtual network interface. With root access to your "server", you can install/configure your VPS the way you want to. You can then easily do traffic accounting directly through your VPS network interface. VPS hosting solution offers a lot more flexibility then a standard shared hosting solution.

PotentialClient
10-16-2004, 09:02 PM
First of all, I would like to thank both of you for responding quickly, I appreciate it. :-)

FH-Dave:

Web, FTP, and Mail traffics are calculated daily by analyzing the log files. You can then see the traffic usage from your control panel.

Traffic is not calculated by SNMP. However, we do have SNMP installed on each web server (if not, you can request it to be installed) and you can then setup cronjob to utilize it.


So, if I understand you correctly, traffic accounting is done only by examining the server log files for the following services:

- mail (smtp(s), pop3(s), imap(s))
- web (http(s))
- ftp

Does that mean that:

- file transfers done via scp do not count towards the monthly data transfer?

- any other type of (ICMP, TCP, UDP) traffic directed at or originating from the dedicated IP assigned to a customer does not count towards the monthly data transfer? No per-interface or per-IP accounting on the server and no IP accounting on the router?

Do you use mrtg+rrd?

Basically, I just want to run a script from a cron job to periodically check the current traffic accounting data and notify me via email if the data transfer gets too close to the monthly limit. I can code the scripts myself, so all I need is confirmation that this can be done on your FreeBSD shared hosting servers.

FH-Dave:

Out of curiousity, have you taken a look at our Virtual Private Server (VPS) solution?

I'm aware of your VPS plans, but your shared hosting plans are better suited to my needs, I would only be hosting a couple of domains for now. Speaking of VPS, do you ever plan on offering jail based FreeBSD VPS plans? That would be something I would be interested in. :-)

FH-Dave
10-16-2004, 11:02 PM
So, if I understand you correctly, traffic accounting is done only by examining the server log files for the following services:

- mail (smtp(s), pop3(s), imap(s))
- web (http(s))
- ftp

Do you use mrtg+rrd?


That is correct. The are the default traffic that's calculated by the control panel. However, the control panel only shows you the total traffic volume on a given billing cycle. It does not give you any data in MRTG graph.


Does that mean that:

- file transfers done via scp do not count towards the monthly data transfer?


SCP works on port 22, so this traffic will not be accounted for by the control panel.


- any other type of (ICMP, TCP, UDP) traffic directed at or originating from the dedicated IP assigned to a customer does not count towards the monthly data transfer? No per-interface or per-IP accounting on the server and no IP accounting on the router?


That is correct. I am trying to see how your question will be relevant to us. We do have firewall in place that will prevent any ports, other than what's already been opened, be opened by users. Not all outgoing/incoming traffic to the server will be passed through. Furthermore, customer is not allowed to run any service/daemond as per our AUP. May I ask why you are interested to see these ICMP/UDP traffics?


Basically, I just want to run a script from a cron job to periodically check the current traffic accounting data and notify me via email if the data transfer gets too close to the monthly limit. I can code the scripts myself, so all I need is confirmation that this can be done on your FreeBSD shared hosting servers.


Can you elaborate on how you will get the traffic data from? I will make sure once again with my admin if he will allow SNMP be installed on the server. Other than SNMP, I am not aware of a way for customer to get traffic data directly from the interface. We can, however, setup something for you and pass the data for your script.

Also, if you are worry about your traffic growing unnoticed (thus, I believe that's why you want to monitor it yourself), you can also limit the traffic from the control panel, for example limit the traffic to 1GB/day, 2GB/week, and et cetera.


I'm aware of your VPS plans, but your shared hosting plans are better suited to my needs, I would only be hosting a couple of domains for now. Speaking of VPS, do you ever plan on offering jail based FreeBSD VPS plans? That would be something I would be interested in. :-)

For what you may want to do, I think the VPS may works much better than shared hosting. There are limitations on shared hosting environment that may prevent you to do things as you proposed above.

Unfortunately, we only offer Linux based VPS at this moment.

PotentialClient
10-17-2004, 12:37 AM
FH-Dave:

That is correct. I am trying to see how your question will be relevant to us. We do have firewall in place that will prevent any ports, other than what's already been opened, be opened by users. Not all outgoing/incoming traffic to the server will be passed through. Furthermore, customer is not allowed to run any service/daemond as per our AUP. May I ask why you are interested to see these ICMP/UDP traffics?


I do not need such detailed traffic statistics, I only wanted to know what counts towards the monthly data transfer because I do not want my traffic to unexpectedly exceed the limit. I have no intention of providing any services on my own or initiating connections other than those required by the CGI scripts to process payments via SSL.

Some hosting companies do IP level accounting (counting even the packets dropped by the firewall!), so some arbitrary Denial of Service attack or a misconfigured search engine bot pounding away on the wrong port can drain one's monthly data transfer allowance very quickly with such accounting policies in place. You can rest assured that the sites I intend to host are not the kind that would draw negative attention or make someone want to launch a DDoS against the host. Being an admin myself I just want to know in advance what I can expect in such situations, even if they are not very likely to happen.

FH-Dave:

Can you elaborate on how you will get the traffic data from? I will make sure once again with my admin if he will allow SNMP be installed on the server. Other than SNMP, I am not aware of a way for customer to get traffic data directly from the interface. We can, however, setup something for you and pass the data for your script.


I just want access to the traffic accounting data that you use for billing, so if it is available through the control panel, then perhaps the simplest way would be to use a Perl script that would basically log into the control panel and fetch the page with the traffic accounting data via curl or wget and then parse it to get the individual traffic counter values. How does that sound to you?

As for SNMP, it would require installing and configuring net-snmp (net-mgmt/net-snmp in the ports tree), which would be quite a hassle in itself, so I think that option should only be considered if the script based solution I just mentioned is not possible.

FH-Dave:

Also, if you are worry about your traffic growing unnoticed (thus, I believe that's why you want to monitor it yourself), you can also limit the traffic from the control panel, for example limit the traffic to 1GB/day, 2GB/week, and et cetera.


You are correct, my main concern is not to exceed the monthly data transfer allowance in any way. It's good to know customers can set their own traffic limits like that. What happens when the customer self-imposed limit is reached? Is incoming email rejected? Does the web server display an error message informing the visitor that the customer whose site they just visited has used up their traffic allowance?

Dave-FH:

For what you may want to do, I think the VPS may works much better than shared hosting. There are limitations on shared hosting environment that may prevent you to do things as you proposed above.


Trust me, the sites I intend to host are very basic and low-volume, but since I'm a very picky customer and a perfectionist in general, I need to know all these details that may make it seem like I need to run some complex, high-volume sites. :-)

FH-John
10-17-2004, 01:03 AM
The control will send you an e-mail notice as you approach the bandwidth limit on your account. It also does the same when you approach your disk allocation.

FH-Dave
10-17-2004, 09:53 AM
FH-Dave:
I just want access to the traffic accounting data that you use for billing, so if it is available through the control panel, then perhaps the simplest way would be to use a Perl script that would basically log into the control panel and fetch the page with the traffic accounting data via curl or wget and then parse it to get the individual traffic counter values. How does that sound to you?


You have access to your own web/http logs. Thus, you can create a script that analyze the log files and give you real-time http monitoring. However, FTP and Mail logs are not placed under your acount. The Mail server itself is located on a separate dedicated server. Same goes with the control panel server and the database server.

So it seems you can only anaylze your http traffic, which, depending on your needs, may be good enough considering http traffic usually takes 90%+ of the total traffic.


You are correct, my main concern is not to exceed the monthly data transfer allowance in any way. It's good to know customers can set their own traffic limits like that. What happens when the customer self-imposed limit is reached?


You can only throttle your http traffic. Thus depending on how you define your traffic limit, the web server will delay requests to your websites in order to meet your traffic shaping definition (e.g. 1 GB per day, etc). This is done by apache module, mod_throttle.


Is incoming email rejected? Does the web server display an error message informing the visitor that the customer whose site they just visited has used up their traffic allowance?


Even if you do not throttle your http traffic, your websites and emails will still function eventhough you are over your traffic limit. You will simply be charged for the over-usage traffic at the rate of $2/GB. And as John has mentioned, when you have reached around 80% of your traffic quota, you will receive an email from the control panel to let you know about this.