[PacketFence-devel] Pfbandwidth Bottleneck

Discussion:

g***@tonarchiv.ch

2016-10-30 09:04:45 UTC

Hi there,

We are using PF in in-band mode and also have accounting enabled.

Now it seems that pfbandwidth is the bottleneck in this setup.
We have 4 core CPU, 8GB RAM and SSD disk. I/O latency and RAM usage is
no issue. But pfbandwidth takes 100% of one core when we put some load
on the box (10 clients, total of approx. 10MB/s).

I understand that pfbandwidth has to be single threaded because of the
way how it gets the traffic data from the kernel.
But I wonder if there is some way to tune it to get a better performance
under high load?

Best regards,
Till

Durand fabrice

2016-11-04 23:27:29 UTC

Permalink

Hello Till,

We probably can do something and save the stats via pfqueue
(inline_accounting_update_session_for_ip in pfqueue).

It need a little bit of dev (really few).

also can you check one the db side if there is no issue ?

Regards

Fabrice

Post by g***@tonarchiv.ch
Hi there,
We are using PF in in-band mode and also have accounting enabled.
Now it seems that pfbandwidth is the bottleneck in this setup.
We have 4 core CPU, 8GB RAM and SSD disk. I/O latency and RAM usage is
no issue. But pfbandwidth takes 100% of one core when we put some load
on the box (10 clients, total of approx. 10MB/s).
I understand that pfbandwidth has to be single threaded because of the
way how it gets the traffic data from the kernel.
But I wonder if there is some way to tune it to get a better performance
under high load?
Best regards,
Till
------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
_______________________________________________
PacketFence-devel mailing list
https://lists.sourceforge.net/lists/listinfo/packetfence-devel

g***@tonarchiv.ch

2016-11-05 12:33:57 UTC

Permalink

Hello Fabrice,

Ah you think that also the DB could be the bottleneck? Hard drive I/O
isn't - latency is still under 1ms and buffers are not filling up.

I guess I need some more fine grained monitoring the understand what is
going on...

Any advice on how to monitor which process is eating CPU? With htop it
always is pfbandwidth on the top line of CPU time use.

I was think that, because it is only one CPU core loaded to 100%, it
must be a single thread. You think that DB handling of the pfbandwidth
thread could be the time consuming part?
Some buffering with a queue sounds like a good solution then.

Thank you,
Till

Post by Durand fabrice
Hello Till,
We probably can do something and save the stats via pfqueue
(inline_accounting_update_session_for_ip in pfqueue).
It need a little bit of dev (really few).
also can you check one the db side if there is no issue ?
Regards
Fabrice

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
PacketFence-devel mailing list
https://lists.sourceforge.net/lists/listinfo/packetfence-devel

Durand fabrice

2016-11-09 00:54:20 UTC

Permalink

Hello Till,

on my side i am using sysdig on order to check what is wrong on the
system and it really help to understand where is the bottleneck.

Also in the next few week i will probably work on pfbandwithd, so i will
add the usage of pfqueue.

Regards

Fabrice

Post by g***@tonarchiv.ch
Hello Fabrice,
Ah you think that also the DB could be the bottleneck? Hard drive I/O
isn't - latency is still under 1ms and buffers are not filling up.
I guess I need some more fine grained monitoring the understand what is
going on...
Any advice on how to monitor which process is eating CPU? With htop it
always is pfbandwidth on the top line of CPU time use.
I was think that, because it is only one CPU core loaded to 100%, it
must be a single thread. You think that DB handling of the pfbandwidth
thread could be the time consuming part?
Some buffering with a queue sounds like a good solution then.
Thank you,
Till

g***@tonarchiv.ch

2016-11-09 20:00:59 UTC

Permalink

Hello Fabrice,

thanks a lot for your help. I just started to use sysdig. I wonder if
there's anything this tool can not measure... ;)

This would be great if you could add the queue usage. I know that
in-band mode is not the preferred solution. But there are situations
where it is the better match. And it is kind of plug and play, which can
be very useful.

I will also replace MySQL by Percona and see if there is a difference ...

Regards,
Till

Post by Durand fabrice
Hello Till,
on my side i am using sysdig on order to check what is wrong on the
system and it really help to understand where is the bottleneck.
Also in the next few week i will probably work on pfbandwithd, so i will
add the usage of pfqueue.
Regards
Fabrice