We started producing shows as Today with a Techie on 2005-09-19, 13 years, 11 months, 12 days ago. Our shows are produced by listeners like you and can be on any topic that "are of interest to Hackers". If you listen to HPR then please consider contributing one show a year. If you record your show now it could be released in 11 days.

Meet the team

Please help out tagging older shows !

In this episode we'll talk about filtering and dissecting packet traces and streams and introduce diffing. Remember that most tools have very flexible options for a variety of use cases. So check their manpages. Each man page also has multiple examples of how to use each tool.

Counting Packets

• Lets start with grabbing a trace from the unit tests:

   $ mkdir /tmp/packets
   $ cd /tmp/packets
   $ cp /path/to/onics/tests/data/packets/sample.xpkt .

• Lets see what we have inside. First, lets see how many packets there are. We'll use a new tool 'pcount'.

   $ pcount sample.xpkt

   90 total packets and 19082 total bytes.

• Good thing we looked first. Don't want to walk through all the packets.

Scanning Packet Flows

• Well, lets look at the connections or "flows" in the trace. We'll do this by using the 'nftrk' command for "network flow tracker".

• Like 'pcount' this utility (and many or most ONICS utilities), this program can run on a live stream or a trace file. We'll run:

   $ nftrk -dt sample.xpkt | grep END

and get:

   |FLOW END|IP:ca=192.168.0.43,sa=224.0.0.251,proto=2|Start=1565446184.543,
   End=1565446184.544,Dur=0.001|SENT:1,60|
   ...
   |FLOW END|IP:ca=192.168.0.7,sa=192.168.0.255,proto=17,cpt=631,spt=631|
   Start=1565446184.543,End=1565446184.544,Dur=0.001|SENT:3,660|

• 'nftrk' tracks flows giving events like the start and end of each flow or connection. We just want a summary of all the connections so we just grep for 'END' (all caps).

• We could just as easily have grepped for START, but this way we get the final number of packets sent and received on each connection. If we just want a count of the connections we can do:

   $ nftrk -dt sample.xpkt | grep START | wc -l

and that tells us that there are 10 flows in the trace.

Basic Filtering

• Ok, so 90 packets, in 10 flows totalling ~19000 bytes. Lets now see about filtering the connection so we just get the TCP packets.

   $ pflt tcp sample.xpkt tcponly.xpkt

   $ pcount tcponly.xpkt
   73 total packets and 17184 total bytes.

   $ nftrk -dt tcponly.xpkt | grep END | wc -l
   2

• We could have been super fancy and done:

   $ pflt tcp sample.xpkt |
     pcount -p |
     nftrk -t 2>/tmp/flows > tcponly.xpkt &&
     echo -n "Number of flows " &&
     grep END /tmp/flows | wc -l &&
     rm -f /tmp/flows

• Ok, enough of that. Anyway, now we have a trace file with only the TCP connections. Running

   $ nftrk -dt /tmp/tcponly.xpkt | grep END
   |FLOW END|IP:ca=192.168.0.4,sa=192.168.0.7,proto=6,cpt=38859,spt=22|
   Start=1566073862.612,End=1566073862.613,Dur=0.000|C2S:25,4561|S2C:30,5124|
   |FLOW END|IP:ca=192.168.0.4,sa=64.233.169.147,proto=6,cpt=35071,spt=80|
   Start=1566073862.613,End=1566073862.613,Dur=0.000|C2S:9,704|S2C:9,6795|

Shows that the server ports are 22 and 80 for the two connections. That's SSH and HTTP.

• The patterns we can use to filter packets are pretty standard across most of the ONICS tools.

• We'll discuss this is more detail in a future podcast. But if you want to see the kinds of fields you can match on go to

   $ man onics_proto

Extracting Ranges of Packets

• What if we wanted to just grab specific packets out of the trace file? Say we wanted packets 3-6. For that we would run:

   $ pxtr 3,6 sample.xpkt pkts-3-to-6.xpkt

• Alternately we could ask for all packets from the 7th packet to the first TCP packet. We match using the same types of matching conditions as with pflt, but we must enclose them in {}s.

   $ pxtr "7,{tcp}" sample.xpkt | xpktdump

• Lets say we just wanted to drop packets 5-10 from the stream. There are several ways to do this in ONICS, but using pxtr, the way we would do it would be:

   $ pxtr 1,4 sample.xpkt > not-5-to-10.xpkt
   $ pxtr 11,NONE sample.xpkt >> not-5-to-10.xpkt

• Maybe I should add another option to pxtr to invert the boundary conditions. It's a tradeoff between having the tools do one thing and one thing well and supporting a potentially common use case.

Differences Between Traces

• Finally, lets look at one tool that I really like. Let's see the difference between the original stream and the one that we just created:

   $ pdiff sample.xpkt not-5-to-10.xpkt | less

• Sure enough that shows us that packets 5-10 were dropped from the stream. If we do the reverse

   $ pdiff -v not-5-to-10.xpkt sample.xpkt | less

it describes the sample.xpkt from the perspective of starting with not-5-to-10.xpkt and inserting a bunch of packets into the middle.

Conclusion

• In this podcast we looked at a few tools to help analyze and dissect packet traces or packet streams.
• Next time we'll look at some of the more powerful pattern matching we can apply and

I discuss and demonstrate the latest retro gadget I found at the flea market last weekend, a TASCAM Porta 02 MiniStudio 4-Track Cassette Recorder. It was in a bin full of junk—filthy, lacking its power supply, and I got it for only $5. I hacked a power supply, disassembled it completely, washed everything thoroughly, and put it back together. It worked perfectly with the exception of the pause button. This has been one of the most fun projects I can remember, especially because my daughter is into it too, and she's learning how to make multi-track recordings. I always wanted a 4-track when I was in high school but never had one. Now I do!

Links to info about stuff I mentioned

• Flickr album with photos from this recording session
• Flickr album with photos from the TASCAM machine restoration
• My earlier episode about the Marantz recorder: hpr1844 :: The Marantz PMD 660 Professional Solid State Recorder
• My video of the TASCAM testing, power supply hack, disassembly, and cleaning: Watch on YouTube
• Multitrack recording
• Güiro (mine is metal)
• Clave (instrument) and clave rhythm (you heard the 2-3 clave rhythm on this podcast)
• Diatonic harmonica
• Chromatic harmonica
• Microphone types
• Phantom power
• Clipping
• EQ: Equalization
• Mixing
• Mastering
• Demo Recordings
• The TASCAM PortaStudio
• David Mead (his first two albums were with RCA, not Polygram
• My first test recording on the restored TASCAM Porta 02: Listen on Soundcloud

For more information, have a look at https://boardgamegeek.com/boardgame/2593/pass-pigs

Background

• It's been about 6 years since I talked about my project ONICS in HPR 1350
• ONICS stands for Open Network Inpection Command Suite
• I created ONICS as because I thought it would be neat to have a suite of tools that could manipulate packets on the command line in a way similar to how tools lik sed, awk, grep, cut, and so forth manipulate text.

Installing

• Not currently maintained in any package distributions
• Maintainers who are interested in doing so are welcome
• Install by source

    $ git clone https://gitlab.com/catlib/catlib
    $ cd catlib
    $ make
    $ cd ..
    $ git clone https://gitlab.com/onics/onics
    $ cd onics
    $ ./configure
    $ make
    $ make test
    $ sudo make install
    $ make veryclean

• Can always uninstall cleanly from the source directory

    $ make uninstall

• Alternate to installation is to stop at 'make test' and then add to 'onics/bin' and 'onics/scripts' to your path.

Documentation

• Manpages are available in onics/doc directory if you aren't installing locally. They are quite extensive.

• If installed locally, starting with:

    $ man onics

XPKT Format

• PCAP format is outdated and not very extensible

  • I want to be able to annotate with interface IDs, flow IDs, packet numbers, classification info, header offsets, etc...

• First and foremost, the file header prevents just cating files together.

  • it makes merging live streams more difficult
  • pcapng improves things but still has global file header

First Programs

• Let's first capture in the traditional way

    $ sudo tcpdump -i eth0 -c 5 -w file1.pcap

• First program is to capture packets from the wire:

    $ sudo pktin eth0 > file2.xpkt

• If not running as root

    $ sudo chown myname file1.pcap file2.xpkt

• Let's dump them:

    $ tcpdump -r file1.pcap
    $ xpktdump file2.xpkt

• Now lets convert the PCAP to XPKT

    $ pc2xpkt file1.pcap file1.xpkt
      or
    $ pc2xpkt file1.pcap > file1.xpkt
      or
    $ pc2xpkt < file1.pcap > file1.xpkt
      or
    $ cat file1.pcap | pc2xpkt > file1.xpkt

• Now we can dump file1 using xpktdump:

    $ xpktdump file1.xpkt

Something we can't do w/ tcpdump

• Lets now merge them one after another

    $ cat file1.xpkt file2.xpkt > merged.xpkt
    $ xpktdump merged.xpkt

• Of course there's a simpler way

    $ cat file1.xpkt file2.xpkt | xpktdump

Convert back to pcap:

• Let's convert file2 to PCAP

    $ xpkt2pc file2.xpkt file2.pcap
      or
    $ xpkt2pc < file2.xpkt > file2.pcap
      or
    $ xpkt2pc file2.xpkt > file2.pcap
      or
    $ cat file2.xpkt | xpkt2pc > file2.pcap

• Let's look at the stream using tcpdump:

    $ tcpdump -r file2.pcap

• If we didn't want to actually store as a PCAP

    $ xpkt2pc file2.xpkt | tcpdump -r -

• Let's concatenate and dump using tcpdump

    $ cat file1.xpkt file2.xpkt | xpkt2pc | tcpdump -r | less

Sending packets:

    $ sudo tcpdump -i eth0  # in one terminal
    $ sudo pktout -i eth0 file1.xpkt
      or
    $ sudo pktout -i eth0 < file1.xpkt
      or
    $ cat file1.xpkt | sudo pktout -i eth0

Summary

• XPKT is a versatile, extensible, self-contained packet trace format
• ONICS' most basic tools are pktin, pktout, pc2xpkt and xpkt2pc
• We've demonstrated how the ONICS design supports leveraging the power of the UNIX command line for packets
• This is only the VERY beginning. ONICS has over 20 binaries and 30 scripts for manipulating packets.

In this show Ken, recalls hpr1771 :: Audacity: Label Tracks by Jon Kulp to add Labels to an large audio file.

• Tidy up the audio to the point where you are happy with it, but do not truncate silence.
• Find the first break in the audio and check how long it is. In my case it was 4 seconds.
• Select the entire track and select Analyze>Silence Finder
• Change Maximum duration of silence to just under the length of the break. In my case I set it to 3 seconds
• This will then create a series of labels on a new Label track
• Edit the names of each as desired.
• Select File > Export > Export Multiple
• Select Split Files based on Labels
• Name files using Label/Track Name

We take the ideas we have developed over the previous episodes and use them to evaluate a a study I found online. These are things anyone can do with just a little work on Google, and the payoff is to have a good idea of whether or not you are looking at a quality study

In this series I cover how I listen to podcasts and how the process has change over the years. This episode badly covers the console audio player moc.

Background

class  Eq a where
  (==) :: a -> a -> Bool
  (/=) :: a -> a -> Bool

  x /= y = not (x == y)

data Size = Small | Medium | Large
    deriving (Show, Read)

instance Eq Size where
    Small == Small = True
    Medium == Medium = True
    Large == Large = True
    _ == _ = False

> Small == Small
True
> Large /= Large
False

data Size = Small | Medium | Large
    deriving (Show, Read, Eq)

class Eq a => Ord a where
    compare :: a -> a -> Ordering
    (<) :: a -> a -> Bool
    (>=) :: a -> a -> Bool
    (>) :: a -> a -> Bool
    (<=) :: a -> a -> Bool
    max :: a -> a -> a
    min :: a -> a -> a

instance Ord Size where
    Small <= _ = True
    Medium <= Small = False
    Medium <= _ = True
    Large <= Large = True
    Large <= _ = False

check :: (Ord a, Show a) => a -> a -> String
check a b =
    case compare a b of
        LT ->
            show a ++ " is smaller than " ++ show b
        GT ->
            show a ++ " is greater than " ++ show b
        EQ ->
            show a ++ " and " ++ show b ++ " are equal"

> check Medium Small
"Medium is greater than Small"
> check Small Large
"Small is smaller than Large"
> check 7 3
"7 is greater than 3"
> check [1, 2] [1, 1, 1]
"[1, 2] is greater than [1, 1, 1]"

Overview

This recipe has been heavily adapted from one I received from Hello Fresh - credit where credit's due!