Project - Noise Calibration
Mentor - Stanislav Shalunov
Weekly Update 1/10
Developer - Cesar Marcondes

Task 1) (June 24th - July 14th)
   Writing some kernel hooks to obtain information
   of the systems

       [Completed]
       - FreeBSD kernel hook (in src/sys/net/if_ethersubr.c)
       Logging the dispersion difference of back to back packets
       using rdtsc (it's possible to monitor the coalescing
       under high load)
       
       [In progress]       
       - Linux kernel hook coming soon
          
       [Completed]
       - Some literature review and reading some papers

   Starting tests and collecting datasets using
   different environments (machines and OS)

       [In progress]
       Emails exchange going on in the Internet2
       context.

       [In progress]
       I talked to some guys here in my university
       and it seems that I may possibly use their
       testbed to perform most of these tests
       (it's a gigabit testbed w/ Xeon 3.02Ghz machines)

Task 2) (July 14th - August 10th)
   Extensive statistical analysis and decisions
   about more tests to be done
      	Develop the statistical model and other
   	statistical tools

Task 3) (August 10th - September 1st)
   Development of the filtering mechanism in a linux
   kernel implementation

----- Attached the original proposal

Attack Plan:
The main goal of this project is to "filter out
noise" from machine delay measurements in a very
robust manner, such that delay-based congestion
control algorithms could work better [Design Space.
pg 6] and a bulk transport can take advantage of
this to better determine the nature of the delay
variance. Therefore, my plan starts with an
extensive set of tests (in the Internet2 or on a
network emulator at Gigabit speeds) to determine
statistically the nature and amplitude of such noise
in different environments. Some recent work by
Dovrolis et al (PAM 2004) observed how the NIC
interrupt coalescing results in noise on capacity
estimation measurements, in this proposal, I'm
particularly interested in measuring the amount of
noise introduced by other OS parts, like context
switches, I/O interruptions, and variation in packet
processing paths as possible machine noise
indicators.

From the determined indicators, I will derive a
sound statistical bayesian model. Once this model
is derived, it should be used to infer robustly the
nature of the noise such that effective filtering
can be done. In such task, I will create some R
scripts and other scripts to automatically process
these massive datasets. (Scripts would be available
as open source at the end of the project) -
Reference: Graphical Models using R

The next step would be a Linux user/kernel
implementation of the following idea:
Build a filtering mechanism that drops/gravitation
predicted bad delay samples (the ones that are drifted
by machine noise), such prediction would be based on
the level of indicators that are constantly monitored and
constantly aggregated statistically. Such "prediction"
will be based on the updated probability from the developed
bayesian model and the observations (data).
(Source code of this filter algorithm will be available
as open source at the end of the project)