Re: tcptrace and huge files ? (a few GB)

From: Jose Manuel Cano Garcia (cano@pc2072te.dte.uma.es)
Date: 10/22/02 

From: Jose Manuel Cano Garcia <cano@pc2072te.dte.uma.es>
Subject: Re: tcptrace and huge files ? (a few GB)
Date: Tue, 22 Oct 2002 19:52:25 +0200
Message-Id: <200210221952.25745.cano@pc2072te.dte.uma.es>

Hi,

tcptrace have memory problems when procesing large files. I think the best
solution is to prefilter the trace with tcpdump.

I had a similar problem with large traces (300 MB) and I did the followings
things:

1) Filter with tcpdump to different files as a function of the service port
(FTP, HTTP, SMTP, others).

2) If the files are still long, (i.e. HTTP) filter following an arbitrary
criteria to clasify connections and further divide the trace in differents
files (i.e. even or odd client port). Of course, care must be taken to avoid
sending to different files packets belonging to the same connection.

I made a script that splits the trace into one non-http and two (clasified
according to the arbitrary parity criteria) http traces, and processes them
separatelly with tcptrace. I enclose it with this mail. Be aware about the
fact that It processes UNCOMPRESSED files only and generates zip compressed
files.

If you want to process such a large file as you say, i think you should split
the http trace into more than two pieces.

Best regards

Jose Manuel

PD: I am interested on traffic measurements. Do you have your traces
available?

El Mar 22 Oct 2002 13:57, Olivier M. escribió:
> Hello,
>
> I'm working on a swiss internet analysis project, and
> we'd have to analyse really big tcpdumps raw files of data
> (2 files of around 5GB, and one of around 46 GB, everything gziped,
> over a 24h time period).
>
> We would have to get statistics on these points:
> - session duration (SYN to FINACK)
> - session throughput histogram (also by daytime)
> - session packet loss
> - session packet retransmits
> - session average RTT
> - session RTT variance
> - session aborts
> - session timing and bursts
> - session in/outbound delay-bandwidth product
> - session MSS changes
> - session flow control characteristics (window sizes)
> - session congestion control
> - session hop count
> and tcptrace would surely help for some of them.
>
> But will it handle so big files correctely ? Have you already
> had some experiences, or so you know better ways to do that?
>
> Thanks in advance for your hints :)
> Olivier Müller

----------------------------------------------------------------------------
To unsubscribe, send a message with body containing "unsubscribe tcptrace" to
majordomo@tcptrace.org.

This archive was generated by hypermail 2b30 : 10/23/02 EDT