#+STARTUP: overview indent
#+html: <p>Linux Branch: "iptfs": <a href="https://github.com/LabNConsulting/iptfs-linux/actions"><img src="https://github.com/LabNConsulting/iptfs-linux/actions/workflows/ci.yml/badge.svg?branch=iptfs"></a>
#+html: <a href="https://codecov.io/gh/LabNConsulting/iptfs-linux" ><img src="https://codecov.io/gh/LabNConsulting/iptfs-linux/branch/iptfs/graph/badge.svg?token=JAO48ACBPT"></a>
#+html: </p>

* Linux TFS development environment

There's a Makefile that will git clone the needed repos.

** Usage Notes

*** Building

If starting from scratch run these 2 commands:

#+begin_src shell
    make setup
    make
#+end_src

This will checkout a linux kernel source tree under ~linux~, a tagged release
of buildroot under ~buildroot~, and a modified branch of iproute2 under
~iproute2~. Then it will build the kernel and the rootfs.

*** Launching a "Host <-> Router <-> Router <-> Host" setup.

You can use the tests to bring up a working iptfs ipsec tunnel. You'll need to
have installed ~qemu~, ~socat~, ~python~ and optional but highly recommended
~tmux~, additionally you'll need to install the python requirements,

#+begin_src bash
  python3 -m venv venv
  source venv/bin/activate
  pip install -r python-requirements.txt
#+end_src

Then run a test adding the `--pause` flag, and it will pause before running the first
test, but after having configured the hosts (h1, h2) and routers (r1, r2).

NOTE: If your `sudo` command forces a restricted PATH (~secure_path~) then the
python virtual environment may not work. In this case `sudo bash` and then
activate the virtual environment as root.

NOTE: SUDO: For best results add the following to your ~/etc/sudoers~ config.
This will allow tmux to continue to work inside the sudo environment.

#+begin_src shell
  Defaults env_keep += "TMUX"
  Defaults env_keep += "TMUX_PANE"
#+end_src

#+begin_src shell
  $ sudo -E pytest -s -v tests/simplenet --pause
  [...]
  == PAUSING: before test 'tests/simplenet/test_simplenet.py::test_net_up' ==
#+end_src

or

#+begin_src shell
  $ sudo -E bash
  # tmux # optional but highly useful
  # source venv/bin/activate
  # pytest -s -v tests/simplenet --pause
  [...]
  == PAUSING: before test 'tests/simplenet/test_simplenet.py::test_net_up' ==
#+end_src

You can now log into the running setup in another terminal. Use ~socat~ to log
into the console of the running qemu'd linux.

#+begin_src shell
  $ sudo socat /dev/stdin,rawer,escape=0x1d,,echo=0,icanon=0 \
      unix-connect:/tmp/unet-test/tests.simplenet.test_simplenet/r1/s/console2
#+end_src

You can use ~mucmd~ to simply enter the namespace of the running node (e.g., to
ping from ~h1~ to ~h2~ over the iptfs tunnel use the following command).

#+begin_src shell
  $ sudo mucmd -d /tmp/unet-test/tests.simplenet.test_simplenet h1 ping 10.0.2.4
  PING 10.0.2.4 (10.0.2.4) 56(84) bytes of data.
  64 bytes from 10.0.2.4: icmp_seq=1 ttl=62 time=1.22 ms
  64 bytes from 10.0.2.4: icmp_seq=2 ttl=62 time=1.40 ms
  64 bytes from 10.0.2.4: icmp_seq=3 ttl=62 time=1.25 ms
  ...
#+end_src

*** Qemu

**** consoles

3 serial consoles are created using unix sockets:
   - '%RUNDIR%/s/console'
   - '%RUNDIR%/s/vcon0'
   - '%RUNDIR%/s/vcon1'
   -
These can be accessed with::

~socat /dev/stdin,escape=0x1d,rawer unix-connect:%RUNDIR%/s/console~

Where if the node is named ~r1~ then ~%RUNDIR%~ is usually /tmp/unet-root/r1/ so:

~sudo socat /dev/stdin,rawer,escape=0x1d,,echo=0,icanon=0 unix-connect:/tmp/unet-root/r1/s/console~

If your ~socat~ doesn't support ~rawer~ option replace with ~raw,echo=0,icanon=0~.

**** GDB
#+begin_src bash
  $ sudo gdb linux/vmlinux
  (gdb) target remote /tmp/unet-root/r1/s/gdbserver
  ...

  or
  (gdb)
  target remote /tmp/unet-test/tests.simplenet.test_simplenet/r1/s/gdbserver
  target remote /tmp/unet-test/tests.simplenet.test_simplenet/r2/s/gdbserver

  target remote /tmp/unet-test/tests.errors.test_errors/r1/s/gdbserver
  target remote /tmp/unet-test/tests.errors.test_errors/r2/s/gdbserver

  target remote /tmp/unet-test/tests.frags.test_frags/r1/s/gdbserver
  target remote /tmp/unet-test/tests.frags.test_frags/r2/s/gdbserver

  target remote /tmp/unet-test/tests.phynic.test_simplenet/r1/s/gdbserver

  target remote /tmp/unet-test/tests.stress.test_stress/r2/s/gdbserver

  target remote /tmp/unet-test/tests.utpkt.test_utpkt/r1/s/gdbserver

  target remote /tmp/unet-test/tests.verify.test_verify/r1/s/gdbserver
#+end_src

** Building/Editing Notes
*** CCLS

I like to use ccls with LSP mode in emacs, which will allow you to jump to all
references with semantics (i.e., it's not just a tag search) as well as
displaying documentation and tab completion of symbols etc..

In order for this to work seemlessly you need to do a couple steps:

  1. cd linux # kernel source directory
  2. run ./scripts/clang-tools/gen_compile_commands.py -d ../output-linux
  3. printf "%%compile_commands.json\n-D__IN_CCLS__\n" > .ccls

** CI Usage
*** Testing.

The tests contained herein are used directly in the ~iptfs-linux~ github project
in it's CI for testing kernel changes.

*** Releases

To create a new release (for artifacts) of a kernel and rootfs on github add a tag.
These artifacts are used by other external project (e.g., by munet to test munet
VM capabilities).

This works in the ~iptfs-dev~ directory as well as in ~iptfs-linux~ as they both
use this repo for CI.

** Design Notes
*** Config

The most basic configuration is to select "iptfs" as the mode rather than
"tunnel" when configuring an IPsec SA.

Additionally there are further configuration options based on the mode the iptfs
tunnel should run in.

**** common configuration

- iptfs-dont-fragment :: boolean to disable fragmenting inner packets, defaults to false
- iptfs-max-queue-size :: The maximum queue size for pending packets to send.
- iptfs-pkt-size :: the size of the outer packet (outer ip, esp, iptfs, + inner ip
  packets), either a value or auto to use PMTU
- iptfs-reorder-window-size :: The number of packets to hold waiting for re-ordered
  packets to arrive before considering those missing packets dropped. Default
  for fixed-rate send is 1. Default for demand-rate is 3. If the other endpoint
  is in dont-fragment mode then this value can be set to 0.
- iptfs-drop-time :: The number microseconds to wait before considering
  the next in sequence packet as lost.

**** fixed-rate fixed-sized configuration

- iptfs-fixed-rate :: a fixed rate to send outer packets
- iptfs-inner-fixed-rate :: alternate form of fixed rate to sepcify inner packets datarate
- iptfs-max-delay :: alternate config for max-queue-size, which is based on the fixed send rate.
- iptfs-disable-congestion-control :: disble congestion control, should only be used
  when the user is in full administrative control of all paths the tunnel may take.
- iptfs-no-pad-only :: don't send all pad packets (debug option)

**** demand-rate configuration
- iptfs-initial-delay :: amount of time in microseconds to wait before servicing
  the output queue after the initial packet arrives (first in queue). This time
  allows for collecting more packets to take advantage of IPTFS packet aggregation.

*** From Steffen's Mail

[...] look at:

net/xfrm/*
net/ipv4/xfrm*
net/ipv4/esp4*
net/ipv6/xfrm*
net/ipv6/esp6*

> Anything else you think might be useful too would be much appreciated of course.

I think TFS should be a new encapsulation mode. We currently have
tunnel, transport and beet mode (and some odd ipv6 modes). Adding
a tfs_tunnel mode to add all the TFS special stuff would be the
way to go at a first glance. The modes are implemented in:

net/xfrm/xfrm_output.c
net/xfrm/xfrm_input.c

** Bugs

- xfrmi_rcv_cb is looking up xfrm_state from our newly created skb from decaping
  iptfs, but it has not xfrm_state so we panic
  - Need to associate the xfrm_state with new skbs too.. is there a refcnt for this?

* Miscellaneous Development Scratchpad

This is an area for development notes. Please don't consider them current or
relevant, they're just a place to keep notes while developing IPTFS.

** Coverage, missing

- Large Packet Fragmenting Prior to ingress

- No reordering window fast path
  [[file:linux/net/xfrm/xfrm_iptfs.c::iptfs_input_ordered(gro_cells, x, skb);]]
- ECN and DSCP for IPv4 and IPv6
  [[file:linux/net/xfrm/xfrm_iptfs.c::if (x->props.flags & XFRM_STATE_DECAP_DSCP)]]

- multi packet frags for single inner (very large packet)

- drop timer, in progress packet
  [[file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf("dropping in-progress reassemble\n");]]
- drop timer, future saved in queue to output now
  [[file:linux/net/xfrm/xfrm_iptfs.c::count = xtfs->w_savedlen ? __reorder_drop(xtfs, &list) : 0;]]
  [[file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf("receiving ordered list of len %u\n", count);]]

*** Reassembly

- during reassembly:
  - seq < want, drops packet, can only happen when reorder-window == 0
    code coverage shows no code here...
    [[file:linux/net/xfrm/xfrm_iptfs.c::XFRM_INC_SA_STATS(xtfs, IPTFS_INPUT_OLD_SEQ);]]
  - seq > want, (will need to push past window), next fragment
    [[file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf("missed needed seq\n");]]
  - seq == want but no fragment
    - all pad, ok
      [[file:linux/net/xfrm/xfrm_iptfs.c::XFRM_INC_SA_STATS(xtfs, IPTFS_INPUT_ALL_PAD_SKIP);]]
    - not pad, abort and use next fragment
      [[file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf("missing expected fragment\n");]]
  - all fragment into next packet

- not during reassembly
  - fragment start (unexpected), handle next fragment in packet

*** Reordering

- reorder past
  [[file:linux/net/xfrm/xfrm_iptfs.c::return __reorder_past(xtfs, inskb, freelist, fcount);]]
- reorder future: dup of one we've received
  [[file:linux/net/xfrm/xfrm_iptfs.c::/* a dup of a future */]]
- reoder far future (larger that window)
  - need something in saved window that also shifts out during middle of shifting
    [[file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf("send slot0 during shift: %llu", s0seq);]]
  - need something at end of window that shifts into slot0 (will happend from the above).
    [[file:linux/net/xfrm/xfrm_iptfs.c::extra_drops--; /* we aren't dropping what's in slot0 */]]
- was_gso == true in ingress/output collect function
- zero reorder window size
   [[file:linux/net/xfrm/xfrm_iptfs.c::iptfs_input_ordered(x, skb);]]
- free list from iptfs_input_reorder
  [[file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf("freeing list of len %u\n", fcount);]]

*** Ingress (collecting)
- input GSO packets
[[file:linux/net/xfrm/xfrm_iptfs.c::netdev_features_t features = netif_skb_features(skb);]]
*** Ingress (sending)
- rehoming a fraglist
[[file:linux/net/xfrm/xfrm_iptfs.c::static struct sk_buff **iptfs_rehome_fraglist(struct sk_buff **nextp,]]

XXXX DO THIS NEXT
- iptfs_should_fragment
[[file:linux/net/xfrm/xfrm_iptfs.c::iptfs_should_fragment(skb2, mtu, remaining)) {]]
XXXX

** Implementation Catalog

*** Utility
_proto
_seq
__esp_seq

*** State Management
xfrm_iptfs_get_rtt_and_delays
xfrm_iptfs_init_state
xfrm_iptfs_state_destroy
xfrm_iptfs_user_init
xfrm_iptfs_copy_to_user_state

*** Egress (receive) functions
skb_copy_bits_seq
iptfs_alloc_header_skb
iptfs_alloc_skb
iptfs_pskb_extract_seq
iptfs_input_save_runt
__iptfs_iplen
__iptfs_iphdrlen

**** Reassembly
iptfs_complete_inner_skb
_iptfs_reassem_done
iptfs_reassem_abort
iptfs_reassem_done
iptfs_reassem_cont
iptfs_input_ordered

**** Reordering
__vec_shift
__reorder_past
__reorder_drop
__reorder_this
iptfs_set_window_drop_times
__reorder_future_fits
__reorder_future_shifts
iptfs_input_reorder

**** Input
iptfs_drop_timer
xfrm_iptfs_input

*** Ingress (sending) functions
**** From xfrm
iptfs_enqueue
xfrm_iptfs_output_collect

**** Dequeue and send
iptfs_xfrm_output
iptfs_output_prepare_skb
iptfs_first_skb
iptfs_rehome_fraglist
iptfs_should_fragment
iptfs_output_queued
iptfs_delay_timer

** header_len usages:
*** Initializing
**** AH (ah[46].c)
  - ah_init_state():
  - ah6_init_state():

#+begin_src C
  if (x->props.flags & XFRM_STATE_ALIGN4)
      x->props.header_len = XFRM_ALIGN4(sizeof(struct ip_auth_hdr) + ahp->icv_trunc_len);
  else
      x->props.header_len = XFRM_ALIGN8(sizeof(struct ip_auth_hdr) + ahp->icv_trunc_len);
  if (x->props.mode == XFRM_MODE_TUNNEL)
       x->props.header_len += sizeof(struct iphdr);

       x->props.header_len += sizeof(struct ipv6hdr);
#+end_src

**** ESP (esp[46].c)
  - esp_init_state():
  - esp6_init_state():
#+begin_src C
  /* TRANSPORT mode only has ESP header/trailer + crypto, no IP */
  x->props.header_len = sizeof(struct ip_esp_hdr) + crypto_aead_ivsize(aead);
  if (x->props.mode == XFRM_MODE_TUNNEL) {
      /* v4 */
      x->props.header_len += sizeof(struct ipv6hdr);
      /* v6 */
      x->props.header_len += sizeof(struct iphdr);
      /* v4 */
  } else if (x->props.mode == XFRM_MODE_BEET && x->sel.family != AF_INET6)
      x->props.header_len += IPV4_BEET_PHMAXLEN;
  /* v6 */
  else if (x->sel.family != AF_INET6)
      x->props.header_len += IPV4_BEET_PHMAXLEN + (sizeof(struct ipv6hdr) - sizeof(struct iphdr));
  if (x->encap) { switch (encap->encap_type) {
      case UDP_ENCAP_ESPINUDP:
          x->props.header_len += sizeof(struct udphdr);
      case UDP_ENCAP_ESPINUDP_NON_IKE:
          x->props.header_len += sizeof(struct udphdr) + 2 * sizeof(u32);
      case TCP_ENCAP_ESPINTCP:
          /* only the length field, TCP encap is done by the socket */
          x->props.header_len += 2;
  } }
  align = ALIGN(crypto_aead_blocksize(aead), 4);
  x->props.trailer_len = align + 1 + crypto_aead_authsize(aead);
#+end_src

**** ESP Offload (esp[46]_offload.c)
- pcode:
#+begin_src C
  static struct sk_buff *xfrm4_transport_gso_segment(struct xfrm_state *x,
                                                     struct sk_buff *skb,
                                                     netdev_features_t features)
  {
      skb->transport_header += x->props.header_len;
      ops = rcu_dereference(inet_offloads[xo->proto]);
      return ops->callbacks.gso_segment(skb, features);
  }

  static struct sk_buff *xfrm4_beet_gso_segment(struct xfrm_state *x,
                                                struct sk_buff *skb,
                                                netdev_features_t features)
  {
      skb->transport_header += x->props.header_len;
      if (x->sel.family != AF_INET6) {
          if (proto == IPPROTO_BEETPH) {
              struct ip_beet_phdr *ph = (struct ip_beet_phdr *)skb->data;
              skb->transport_header += ph->hdrlen * 8;
              proto = ph->nexthdr;
          } else {
              skb->transport_header -= IPV4_BEET_PHMAXLEN;
          }
      } else {
          __be16 frag;
          skb->transport_header += ipv6_skip_exthdr(skb, 0, &proto, &frag);
          if (proto == IPPROTO_TCP)
              skb_shinfo(skb)->gso_type |= SKB_GSO_TCPV4;
      }

      if (proto == IPPROTO_IPV6)
          skb_shinfo(skb)->gso_type |= SKB_GSO_IPXIP4;
      __skb_pull(skb, skb_transport_offset(skb));
      ops = rcu_dereference(inet_offloads[xo->proto]);
      return ops->callbacks.gso_segment(skb, features);
  }
#+end_src

**** IPCOMP (ipcomp.c)
- pcode:
#+begin_src C
  static int ipcomp4_init_state(struct xfrm_state *x, struct netlink_ext_ack *extack)
  {
      x->props.header_len = 0;
      if (x->props.mode == XFRM_MODE_TUNNEL)
          x->props.header_len += sizeof(struct iphdr);
  }
#+end_src
**** IPIP (xfrm4_tunnel.c)
- pcode:
#+begin_src C
  static int ipip_init_state(struct xfrm_state *x, struct netlink_ext_ack *extack)
  {
      x->props.header_len = sizeof(struct iphdr);
  }
#+end_src
*** Using
**** xfrm_state:
- pcode: xfrm_state_mtu
- This is used as a pad-to target maybe along with configured pad-to size
the padding added to the trailer will be (this value - skb->len)
#+begin_src C
  u32 xfrm_state_mtu(struct xfrm_state *x, int mtu)
  {
      if (x->km.state != XFRM_STATE_VALID || !type || type->proto != IPPROTO_ESP)
          return mtu - x->props.header_len;
      blksize = ALIGN(crypto_aead_blocksize(aead), 4);
      switch (x->props.mode) {
      case XFRM_MODE_TRANSPORT:
      case XFRM_MODE_BEET:
          if (x->props.family == AF_INET)
              net_adj = sizeof(struct iphdr);
          else if (x->props.family == AF_INET6)
              net_adj = sizeof(struct ipv6hdr);
          break;
      case XFRM_MODE_TUNNEL:
      case XFRM_MODE_IPTFS:
          break;
      default:
          WARN_ON_ONCE(1);
          break;
      }
      return ((mtu - x->props.header_len - crypto_aead_authsize(aead) - net_adj) & ~(blksize - 1))
             + net_adj - 2;
      /* TUNNEL is this right what is this value? */
      /* ((1500 - size(IP+ESP) - size(CRYPTAUTH) - 0) & 0xFFFFFFFc) + 0 - 2;

      /* TRANSPORT */
      /* ((1500 - size(ESP) - size(CRYPTAUTH) - size(IP)) & 0xFFFFFFFc) + size(IP) - 2; */
  }

#+end_src
**** xfrm_device
- pcode: prepping skb's for???

This code is going to `skb_pull` `skb_data` down over the mac header and
header_len of data. For IPTFS we want it to leave the IPTFS header in the data
area for encryption.

#+begin_src C

  static void __xfrm_transport_prep(struct xfrm_state *x, struct sk_buff *skb)
  {
      if (xo->flags & XFRM_GSO_SEGMENT)
          skb->transport_header = skb->network_header + sizeof(struct iphdr) || sizeof(struct ipv6hdr);
      pskb_pull(skb, skb_transport_offset(skb) + x->props.header_len);
  }
  static void __xfrm_mode_tunnel_prep(struct xfrm_state *x, struct sk_buff *skb)
  {
      if (xo->flags & XFRM_GSO_SEGMENT)
          skb->transport_header = skb->network_header + sizeof(struct iphdr) || sizeof(struct ipv6hdr);
      pskb_pull(skb, skb->mac_len + x->props.header_len);
  }
  static void __xfrm_mode_iptfs_prep(struct xfrm_state *x, struct sk_buff *skb)
  {
      if (xo->flags & XFRM_GSO_SEGMENT)
          skb->transport_header = skb->network_header + sizeof(struct iphdr) || sizeof(struct ipv6hdr);
      pskb_pull(skb, skb->mac_len + x->props.header_len - sizeof(struct iptfs_header));
  }
  static void __xfrm_mode_beet_prep(struct xfrm_state *x, struct sk_buff *skb)
  {
      if (xo->flags & XFRM_GSO_SEGMENT)
          skb->transport_header = skb->network_header + sizeof(struct iphdr) || sizeof(struct ipv6hdr);
      /* this is reversing something done earlier */
      if (x->sel.family != AF_INET6) {
          phlen = IPV4_BEET_PHMAXLEN;
          if (x->outer_mode.family == AF_INET6)
              phlen += sizeof(struct ipv6hdr) - sizeof(struct iphdr);
      }
      pskb_pull(skb, skb->mac_len + hsize + (x->props.header_len - phlen));
  }
#+end_src

**** Call path for xfrm_outer_mode_prep:
static const struct proto_ops packet_ops = {
	.sendmsg =	packet_sendmsg,
packet_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
packet_snd(struct socket *sock, struct msghdr *msg, size_t len) || tpacket_snd(struct packet_sock *po, struct msghdr *msg)
packet_xmit(const struct packet_sock *po, struct sk_buff *skb)
dev_direct_xmit(struct sk_buff *skb, u16 queue_id)
__dev_direct_xmit(struct sk_buff *skb, u16 queue_id) || sch_direct_xmit(struct sk_buff *skb, struct Qdisc *q,
validate_xmit_skb_list(struct sk_buff *skb, struct net_device *dev, bool *again)
validate_xmit_skb(struct sk_buff *skb, struct net_device *dev, bool *again)
validate_xmit_xfrm
xfrm_outer_mode_prep
**** xfrm_policy

** Offload vs No offload backtrace
**** No ESP offload
#0  esp_output (x=0xffff888107a8a000, skb=0xffff888109584b40) at /home/chopps/w/iptfs-dev/linux/net/ipv4/esp4.c:655
#1  0xffffffff827a5f8c in xfrm_output_one (err=0, skb=0xffff888109584b40) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:600

#2  xfrm_output_resume (sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109584b40, err=<optimized out>, err@entry=1)
    at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:635
#3  0xffffffff827a6483 in xfrm_output2 (skb=0xffff888109584b40, sk=0x0 <fixed_percpu_data>, net=0xffffffff849f7e40 <init_net>)
    at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:662
#4  xfrm_output (sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109584b40) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:826
#5  0xffffffff827b9052 in iptfs_xfrm_output (skb=skb@entry=0xffff888109584b40, remaining=remaining@entry=1358)
    at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:1648
#6  0xffffffff827bc427 in iptfs_output_queued (list=0xffff888237609db0, x=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:2080
#7  iptfs_delay_timer (me=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:2129

#8  0xffffffff812431fe in __run_hrtimer (flags=642, now=0xffff888237609ec0, timer=0xffff888107a8f468, base=0xffff88823762b3c0,
    cpu_base=0xffff88823762b240) at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1685
#9  __hrtimer_run_queues (cpu_base=cpu_base@entry=0xffff88823762b240, now=now@entry=39634970245, flags=flags@entry=642, active_mask=active_mask@entry=240)
    at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1749
#10 0xffffffff8124358b in hrtimer_run_softirq (h=<optimized out>) at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1766
#11 0xffffffff82b50b0c in __do_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:571
#12 0xffffffff8113a70c in invoke_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:445
#13 __irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:650
#14 irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:662
#15 0xffffffff82b3b5ca in common_interrupt (regs=0xffffffff83a07d58, error_code=<optimized out>)
    at /home/chopps/w/iptfs-dev/linux/arch/x86/kernel/irq.c:240
#16 0xffffffff82c0146b in asm_common_interrupt () at /home/chopps/w/iptfs-dev/linux/arch/x86/include/asm/idtentry.h:636
#17 0x0000000000000000 in ?? ()
(gdb)

**** ESP offload (INET_ESP_OFFLOAD)

#0  esp_xmit (x=0xffff888107a8e000, skb=0xffff888109589b40, features=0)
    at /home/chopps/w/iptfs-dev/linux/net/ipv4/esp4_offload.c:257
#1  0xffffffff827bcfd4 in validate_xmit_xfrm (skb=skb@entry=0xffff888109589b40, features=features@entry=1,
    again=again@entry=0xffff888237609658) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_device.c:187

#2  0xffffffff825766db in validate_xmit_skb (skb=skb@entry=0xffff888109589b40, dev=dev@entry=0xffff8881053be000,
    again=again@entry=0xffff888237609658) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3687
#3  0xffffffff82576a1b in validate_xmit_skb_list (skb=0xffff888109589b40, dev=dev@entry=0xffff8881053be000,
    again=again@entry=0xffff888237609658) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3709
#4  0xffffffff82608321 in sch_direct_xmit (skb=skb@entry=0xffff888109589b40, q=q@entry=0xffff8881084e6000,
    dev=dev@entry=0xffff8881053be000, txq=txq@entry=0xffff888104990c00, root_lock=root_lock@entry=0x0 <fixed_percpu_data>,
    validate=validate@entry=true) at /home/chopps/w/iptfs-dev/linux/net/sched/sch_generic.c:327
#5  0xffffffff82577b8d in __dev_xmit_skb (txq=0xffff888104990c00, dev=0xffff8881053be000, q=0xffff8881084e6000,
    skb=0xffff888109589b40) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3805
#6  __dev_queue_xmit (skb=skb@entry=0xffff888109589b40, sb_dev=sb_dev@entry=0x0 <fixed_percpu_data>)
    at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:4210
#7  0xffffffff82590b8d in dev_queue_xmit (skb=0xffff888109589b40)
    at /home/chopps/w/iptfs-dev/linux/include/linux/netdevice.h:3085

#8  neigh_resolve_output (neigh=0xffff8881092ac000, skb=0xffff888109589b40)
    at /home/chopps/w/iptfs-dev/linux/net/core/neighbour.c:1552
#9  0xffffffff826a844d in neigh_output (skip_cache=false, skb=0xffff888109589b40, n=0xffff8881092ac000)
    at /home/chopps/w/iptfs-dev/linux/include/net/neighbour.h:544
#10 ip_finish_output2 (net=net@entry=0xffffffff849fcec0 <init_net>, sk=sk@entry=0x0 <fixed_percpu_data>,
    skb=<optimized out>, skb@entry=0xffff888109589b40) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_output.c:229
#11 0xffffffff826aa7d6 in __ip_finish_output (net=net@entry=0xffffffff849fcec0 <init_net>,
    sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109589b40)
    at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_output.c:307
#12 0xffffffff826aaa72 in ip_finish_output (skb=0xffff888109589b40, sk=0x0 <fixed_percpu_data>,
    net=0xffffffff849fcec0 <init_net>) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_output.c:317
#13 NF_HOOK_COND (pf=2 '\002', hook=4, okfn=0xffffffff826aa910 <ip_finish_output>, cond=<optimized out>,
    out=0xffff8881053be000, in=0xffff8881053aa000, skb=0xffff888109589b40, sk=0x0 <fixed_percpu_data>,
    net=0xffffffff849fcec0 <init_net>) at /home/chopps/w/iptfs-dev/linux/include/linux/netfilter.h:292
#14 ip_output (net=net@entry=0xffffffff849fcec0 <init_net>, sk=sk@entry=0x0 <fixed_percpu_data>,
    skb=skb@entry=0xffff888109589b40) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_output.c:431
#15 0xffffffff827b7bd9 in dst_output (skb=0xffff888109589b40, sk=0x0 <fixed_percpu_data>, net=0xffffffff849fcec0 <init_net>)
    at /home/chopps/w/iptfs-dev/linux/include/net/dst.h:458

#16 xfrm_output_resume (sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109589b40, err=err@entry=1)
    at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:643
#17 0xffffffff827b97dd in xfrm_output2 (skb=0xffff888109589b40, sk=0x0 <fixed_percpu_data>, net=<optimized out>)
    at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:662
#18 xfrm_output (sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109589b40)
    at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:826
#19 0xffffffff827ce922 in iptfs_xfrm_output (skb=skb@entry=0xffff888109589b40, remaining=remaining@entry=1358)
    at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:1648
#20 0xffffffff827d1cf7 in iptfs_output_queued (list=0xffff888237609db0, x=<optimized out>)
    at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:2080
#21 iptfs_delay_timer (me=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:2129


#22 0xffffffff812431fe in __run_hrtimer (flags=642, now=0xffff888237609ec0, timer=0xffff888107a91c68,
    base=0xffff88823762b3c0, cpu_base=0xffff88823762b240) at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1685
#23 __hrtimer_run_queues (cpu_base=cpu_base@entry=0xffff88823762b240, now=now@entry=123664696935, flags=flags@entry=642,
    active_mask=active_mask@entry=240) at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1749
#24 0xffffffff8124358b in hrtimer_run_softirq (h=<optimized out>)
    at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1766
#25 0xffffffff82b67b0c in __do_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:571
#26 0xffffffff8113a70c in invoke_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:445
#27 __irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:650
#28 irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:662
#29 0xffffffff82b525ca in common_interrupt (regs=0xffffffff83a07d58, error_code=<optimized out>)
    at /home/chopps/w/iptfs-dev/linux/arch/x86/kernel/irq.c:240
#30 0xffffffff82c0146b in asm_common_interrupt () at /home/chopps/w/iptfs-dev/linux/arch/x86/include/asm/idtentry.h:636
#31 0x0000000000000000 in ?? ()
(gdb) c

ARP

#0  esp_xmit (x=0xffff888107a8e000, skb=0xffff8881095893c0, features=0)
    at /home/chopps/w/iptfs-dev/linux/net/ipv4/esp4_offload.c:270
#1  0xffffffff827bcfd4 in validate_xmit_xfrm (skb=skb@entry=0xffff8881095893c0, features=features@entry=1,
    again=again@entry=0xffff8882376094a8) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_device.c:187
#2  0xffffffff825766db in validate_xmit_skb (skb=skb@entry=0xffff8881095893c0, dev=dev@entry=0xffff8881053be000,
    again=again@entry=0xffff8882376094a8) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3687
#3  0xffffffff82576a1b in validate_xmit_skb_list (skb=0xffff8881095893c0, dev=dev@entry=0xffff8881053be000,
    again=again@entry=0xffff8882376094a8) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3709
#4  0xffffffff82608321 in sch_direct_xmit (skb=skb@entry=0xffff8881095893c0, q=q@entry=0xffff8881084e6000,
    dev=dev@entry=0xffff8881053be000, txq=txq@entry=0xffff888104990c00, root_lock=root_lock@entry=0x0 <fixed_percpu_data>,
    validate=validate@entry=true) at /home/chopps/w/iptfs-dev/linux/net/sched/sch_generic.c:327
#5  0xffffffff82577b8d in __dev_xmit_skb (txq=0xffff888104990c00, dev=0xffff8881053be000, q=0xffff8881084e6000,
    skb=0xffff8881095893c0) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3805
#6  __dev_queue_xmit (skb=skb@entry=0xffff8881095893c0, sb_dev=sb_dev@entry=0x0 <fixed_percpu_data>)
    at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:4210
#7  0xffffffff82590b8d in dev_queue_xmit (skb=0xffff8881095893c0)
    at /home/chopps/w/iptfs-dev/linux/include/linux/netdevice.h:3085
#8  neigh_resolve_output (neigh=0xffff8881092ac000, skb=0xffff8881095893c0)
    at /home/chopps/w/iptfs-dev/linux/net/core/neighbour.c:1552
#9  0xffffffff82590fa0 in __neigh_update (neigh=neigh@entry=0xffff8881092ac000, lladdr=<optimized out>,
    lladdr@entry=0xffff8881058ff056 "\002\252\252\252\002\003\n", new=<optimized out>, flags=0, nlmsg_pid=nlmsg_pid@entry=0,
    extack=extack@entry=0x0 <fixed_percpu_data>) at /home/chopps/w/iptfs-dev/linux/net/core/neighbour.c:1450
#10 0xffffffff82591c93 in neigh_update (neigh=neigh@entry=0xffff8881092ac000,
    lladdr=lladdr@entry=0xffff8881058ff056 "\002\252\252\252\002\003\n", new=<optimized out>, flags=<optimized out>,
    nlmsg_pid=nlmsg_pid@entry=0) at /home/chopps/w/iptfs-dev/linux/net/core/neighbour.c:1477
#11 0xffffffff8271fbc5 in arp_process (net=net@entry=0xffffffff849fcec0 <init_net>, sk=sk@entry=0x0 <fixed_percpu_data>,
    skb=skb@entry=0xffff888109589640) at /home/chopps/w/iptfs-dev/linux/net/ipv4/arp.c:932
#12 0xffffffff827206d4 in NF_HOOK (pf=3 '\003', sk=0x0 <fixed_percpu_data>, okfn=0xffffffff8271f700 <arp_process>,
    out=0x0 <fixed_percpu_data>, in=0xffff8881053be000, skb=0xffff888109589640, net=0xffffffff849fcec0 <init_net>, hook=0)
    at /home/chopps/w/iptfs-dev/linux/include/linux/netfilter.h:303
#13 arp_rcv (skb=0xffff888109589640, dev=0xffff8881053be000, pt=<optimized out>, orig_dev=<optimized out>)
    at /home/chopps/w/iptfs-dev/linux/net/ipv4/arp.c:988
#14 0xffffffff8257b1bc in __netif_receive_skb_list_ptype (orig_dev=0xffff8881053be000,
    pt_prev=0xffffffff842766c0 <arp_packet_type>, head=0xffff888237609a48)
    at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5539
#15 __netif_receive_skb_list_core (head=head@entry=0xffff8881053c2110, pfmemalloc=pfmemalloc@entry=false)
    at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5582
#16 0xffffffff8257bd97 in __netif_receive_skb_list (head=0xffff8881053c2110)
    at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5634
#17 netif_receive_skb_list_internal (head=head@entry=0xffff8881053c2110)
    at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5725
#18 0xffffffff8257c983 in gro_normal_list (napi=0xffff8881053c2008) at /home/chopps/w/iptfs-dev/linux/include/net/gro.h:433
#19 napi_complete_done (n=n@entry=0xffff8881053c2008, work_done=work_done@entry=1)
    at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:6065
#20 0xffffffff81f70300 in virtqueue_napi_complete (processed=<optimized out>, vq=0xffff888105357800, napi=0xffff8881053c2008)
    at /home/chopps/w/iptfs-dev/linux/drivers/net/virtio_net.c:406
#21 virtnet_poll (napi=<optimized out>, budget=<optimized out>)
    at /home/chopps/w/iptfs-dev/linux/drivers/net/virtio_net.c:1965
#22 0xffffffff8257cc91 in __napi_poll (n=n@entry=0xffff8881053c2008, repoll=repoll@entry=0xffff888237609ed0)
    at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:6496
#23 0xffffffff8257d964 in napi_poll (repoll=0xffff888237609f00, n=0xffff8881053c2008)
    at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:6563
#24 net_rx_action (h=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:6696
#25 0xffffffff82b67b0c in __do_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:571
#26 0xffffffff8113a70c in invoke_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:445
#27 __irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:650
#28 irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:662
#29 0xffffffff82b525ca in common_interrupt (regs=0xffffffff83a07d58, error_code=<optimized out>)
    at /home/chopps/w/iptfs-dev/linux/arch/x86/kernel/irq.c:240
#30 0xffffffff82c0146b in asm_common_interrupt () at /home/chopps/w/iptfs-dev/linux/arch/x86/include/asm/idtentry.h:636
#31 0x0000000000000000 in ?? ()

** nr_frags large packet
(gdb) p *skb
$25 = {{{next = 0x0 <fixed_percpu_data>, prev = 0x0 <fixed_percpu_data>, {dev = 0xffff888109008000, dev_scratch = 18446612686516027392}}, rbnode = {
      __rb_parent_color = 0, rb_right = 0x0 <fixed_percpu_data>, rb_left = 0xffff888109008000}, list = {next = 0x0 <fixed_percpu_data>,
      prev = 0x0 <fixed_percpu_data>}, ll_node = {next = 0x0 <fixed_percpu_data>}}, {sk = 0x0 <fixed_percpu_data>, ip_defrag_offset = 0}, {tstamp = 0,
    skb_mstamp_ns = 0},
  cb = "\b", '\000' <repeats 19 times>, "\001\000\000\000ȉ\373\377\000\000\000\000\316\000\000\000\024\000\000\000ʠ\377\377\342\000\377\377", {{
      _skb_refdst = 18446612686612525184, destructor = 0x0 <fixed_percpu_data>}, tcp_tsorted_anchor = {next = 0xffff88810ec0f080,
      prev = 0x0 <fixed_percpu_data>}, _sk_redir = 18446612686612525184}, _nfct = 0, len = 8040, data_len = 4982, mac_len = 0, hdr_len = 0,
  queue_mapping = 1, __cloned_offset = 0xffff88810e8fb43e "", cloned = 0 '\000', nohdr = 0 '\000', fclone = 0 '\000', peeked = 0 '\000',
  head_frag = 0 '\000', pfmemalloc = 0 '\000', pp_recycle = 0 '\000', active_extensions = 0 '\000', {{__pkt_type_offset = 0xffff88810e8fb440 "",
      pkt_type = 0 '\000', ignore_df = 0 '\000', dst_pending_confirm = 0 '\000', ip_summed = 0 '\000', ooo_okay = 0 '\000',
      __mono_tc_offset = 0xffff88810e8fb441 "", mono_delivery_time = 0 '\000', tc_at_ingress = 0 '\000', tc_skip_classify = 0 '\000',
      remcsum_offload = 0 '\000', csum_complete_sw = 0 '\000', csum_level = 0 '\000', inner_protocol_type = 0 '\000', l4_hash = 1 '\001', sw_hash = 0 '\000',
      wifi_acked_valid = 0 '\000', wifi_acked = 0 '\000', no_fcs = 0 '\000', encapsulation = 0 '\000', encap_hdr_csum = 0 '\000', csum_valid = 0 '\000',
      ndisc_nodetype = 0 '\000', redirected = 0 '\000', nf_skip_egress = 0 '\000', slow_gro = 1 '\001', tc_index = 0, alloc_cpu = 1, {csum = 1048802, {
          csum_start = 226, csum_offset = 16}}, priority = 0, skb_iif = 8, hash = 2159314812, {vlan_all = 0, {vlan_proto = 0, vlan_tci = 0}}, {napi_id = 69,
        sender_cpu = 69}, secmark = 0, {mark = 0, reserved_tailroom = 0}, {inner_protocol = 0, inner_ipproto = 0 '\000'}, inner_transport_header = 0,
      inner_network_header = 0, inner_mac_header = 0, protocol = 8, transport_header = 226, network_header = 206, mac_header = 206}, headers = {
      __pkt_type_offset = 0xffff88810e8fb440 "", pkt_type = 0 '\000', ignore_df = 0 '\000', dst_pending_confirm = 0 '\000', ip_summed = 0 '\000',
      ooo_okay = 0 '\000', __mono_tc_offset = 0xffff88810e8fb441 "", mono_delivery_time = 0 '\000', tc_at_ingress = 0 '\000', tc_skip_classify = 0 '\000',
      remcsum_offload = 0 '\000', csum_complete_sw = 0 '\000', csum_level = 0 '\000', inner_protocol_type = 0 '\000', l4_hash = 1 '\001', sw_hash = 0 '\000',
      wifi_acked_valid = 0 '\000', wifi_acked = 0 '\000', no_fcs = 0 '\000', encapsulation = 0 '\000', encap_hdr_csum = 0 '\000', csum_valid = 0 '\000',
      ndisc_nodetype = 0 '\000', redirected = 0 '\000', nf_skip_egress = 0 '\000', slow_gro = 1 '\001', tc_index = 0, alloc_cpu = 1, {csum = 1048802, {
          csum_start = 226, csum_offset = 16}}, priority = 0, skb_iif = 8, hash = 2159314812, {vlan_all = 0, {vlan_proto = 0, vlan_tci = 0}}, {napi_id = 69,
        sender_cpu = 69}, secmark = 0, {mark = 0, reserved_tailroom = 0}, {inner_protocol = 0, inner_ipproto = 0 '\000'}, inner_transport_header = 0,
      inner_network_header = 0, inner_mac_header = 0, protocol = 8, transport_header = 226, network_header = 206, mac_header = 206}}, tail = 3264,
  end = 3776, head = 0xffff88810ed64000 "", data = 0xffff88810ed640ce "E", truesize = 9334, users = {refs = {counter = 1}},
  extensions = 0x0 <fixed_percpu_data>}
(gdb) p *(struct skb_shared_info *)(skb->head + skb->end)
$26 = {flags = 0 '\000', meta_len = 0 '\000', nr_frags = 2 '\002', tx_flags = 0 '\000', gso_size = 0, gso_segs = 0, frag_list = 0x0 <fixed_percpu_data>,
  hwtstamps = {{hwtstamp = 0, netdev_data = 0x0 <fixed_percpu_data>}}, gso_type = 0, tskey = 0, dataref = {counter = 1}, xdp_frags_size = 0,
  destructor_arg = 0x0 <fixed_percpu_data>, frags = {{bv_page = 0xffffea000432cb00, bv_len = 3072, bv_offset = 192}, {bv_page = 0xffffea000432cb80,
      bv_len = 1910, bv_offset = 192}, {bv_page = 0x0 <fixed_percpu_data>, bv_len = 0, bv_offset = 0} <repeats 15 times>}}

These 2 pages are 0x80 apart (sizeof *page == 0x40 so 2 pages), but the offset
is 192 (0xc0) on both with lengths of 3072 and 1910 these regions overlap.
That's b/c the actual page address is not the page struct.

Here's a SW based 2 frag version:
(gdb) p *skb
$5 = {{{next = 0x0 <fixed_percpu_data>, prev = 0x0 <fixed_percpu_data>, {
        dev = 0xffff888109210000, dev_scratch = 18446612686518157312}}, rbnode = {
      __rb_parent_color = 0, rb_right = 0x0 <fixed_percpu_data>,
      rb_left = 0xffff888109210000}, list = {next = 0x0 <fixed_percpu_data>,
      prev = 0x0 <fixed_percpu_data>}, ll_node = {next = 0x0 <fixed_percpu_data>}}, {
    sk = 0x0 <fixed_percpu_data>, ip_defrag_offset = 0}, {tstamp = 0, skb_mstamp_ns = 0},
  cb = "\004", '\000' <repeats 19 times>, "\001", '\000' <repeats 11 times>, "N\000\000\000\024\000\000\000\306\340\377\377b\000\000", {{_skb_refdst = 18446612686654115968,
      destructor = 0x0 <fixed_percpu_data>}, tcp_tsorted_anchor = {next = 0xffff8881113b9080,
      prev = 0x0 <fixed_percpu_data>}, _sk_redir = 18446612686654115968}, _nfct = 0,
  len = 9000, data_len = 8948, mac_len = 0, hdr_len = 0, queue_mapping = 1,
  __cloned_offset = 0xffff8881112f657e "", cloned = 0 '\000', nohdr = 0 '\000',
  fclone = 0 '\000', peeked = 0 '\000', head_frag = 0 '\000', pfmemalloc = 0 '\000',
  pp_recycle = 0 '\000', active_extensions = 0 '\000', {{
      __pkt_type_offset = 0xffff8881112f6580 "", pkt_type = 0 '\000', ignore_df = 0 '\000',
      dst_pending_confirm = 0 '\000', ip_summed = 0 '\000', ooo_okay = 0 '\000',
      __mono_tc_offset = 0xffff8881112f6581 "", mono_delivery_time = 0 '\000',
      tc_at_ingress = 0 '\000', tc_skip_classify = 0 '\000', remcsum_offload = 0 '\000',
      csum_complete_sw = 0 '\000', csum_level = 0 '\000', inner_protocol_type = 0 '\000',
      l4_hash = 0 '\000', sw_hash = 0 '\000', wifi_acked_valid = 0 '\000',
      wifi_acked = 0 '\000', no_fcs = 0 '\000', encapsulation = 0 '\000',
      encap_hdr_csum = 0 '\000', csum_valid = 0 '\000', ndisc_nodetype = 0 '\000',
      redirected = 0 '\000', nf_skip_egress = 0 '\000', slow_gro = 1 '\001', tc_index = 0,
      alloc_cpu = 1, {csum = 1048674, {csum_start = 98, csum_offset = 16}}, priority = 0,
      skb_iif = 4, hash = 0, {vlan_all = 0, {vlan_proto = 0, vlan_tci = 0}}, {napi_id = 66,
        sender_cpu = 66}, secmark = 0, {mark = 0, reserved_tailroom = 0}, {
        inner_protocol = 0, inner_ipproto = 0 '\000'}, inner_transport_header = 0,
      inner_network_header = 0, inner_mac_header = 0, protocol = 8, transport_header = 98,
      network_header = 78, mac_header = 78}, headers = {
      __pkt_type_offset = 0xffff8881112f6580 "", pkt_type = 0 '\000', ignore_df = 0 '\000',
      dst_pending_confirm = 0 '\000', ip_summed = 0 '\000', ooo_okay = 0 '\000',
      __mono_tc_offset = 0xffff8881112f6581 "", mono_delivery_time = 0 '\000',
      tc_at_ingress = 0 '\000', tc_skip_classify = 0 '\000', remcsum_offload = 0 '\000',
      csum_complete_sw = 0 '\000', csum_level = 0 '\000', inner_protocol_type = 0 '\000',
      l4_hash = 0 '\000', sw_hash = 0 '\000', wifi_acked_valid = 0 '\000',
      wifi_acked = 0 '\000', no_fcs = 0 '\000', encapsulation = 0 '\000',
      encap_hdr_csum = 0 '\000', csum_valid = 0 '\000', ndisc_nodetype = 0 '\000',
      redirected = 0 '\000', nf_skip_egress = 0 '\000', slow_gro = 1 '\001', tc_index = 0,
      alloc_cpu = 1, {csum = 1048674, {csum_start = 98, csum_offset = 16}}, priority = 0,
      skb_iif = 4, hash = 0, {vlan_all = 0, {vlan_proto = 0, vlan_tci = 0}}, {napi_id = 66,
        sender_cpu = 66}, secmark = 0, {mark = 0, reserved_tailroom = 0}, {
        inner_protocol = 0, inner_ipproto = 0 '\000'}, inner_transport_header = 0,
      inner_network_header = 0, inner_mac_header = 0, protocol = 8, transport_header = 98,
      network_header = 78, mac_header = 78}}, tail = 130, end = 320,
  head = 0xffff88810fa37c00 "", data = 0xffff88810fa37c4e "E", truesize = 9844, users = {
    refs = {counter = 1}}, extensions = 0x0 <fixed_percpu_data>}
(gdb) p *((struct skb_shared_info *)(skb->head + skb->end))
$6 = {flags = 0 '\000', meta_len = 0 '\000', nr_frags = 2 '\002', tx_flags = 0 '\000',
  gso_size = 0, gso_segs = 0, frag_list = 0x0 <fixed_percpu_data>, hwtstamps = {{
      hwtstamp = 0, netdev_data = 0x0 <fixed_percpu_data>}}, gso_type = 0, tskey = 0,
  dataref = {counter = 1}, xdp_frags_size = 0, destructor_arg = 0x0 <fixed_percpu_data>,
  frags = {{bv_page = 0xffffea0004387000, bv_len = 3506, bv_offset = 29262}, {
      bv_page = 0xffffea0004387200, bv_len = 5442, bv_offset = 0}, {
      bv_page = 0x0 <fixed_percpu_data>, bv_len = 0, bv_offset = 0} <repeats 15 times>}}

$3 = {
[...]
  frags = {{
      bv_page = 0xffffea0004387000,
      bv_len = 3506,
      bv_offset = 29262
    }, {
      bv_page = 0xffffea0004387200,
      bv_len = 5442,
      bv_offset = 0
    }, {
      bv_page = 0x0 <fixed_percpu_data>,
      bv_len = 0,
      bv_offset = 0
    } <repeats 15 times>}
}

(gdb) p *$7->frags[0].bv_page
$18 = {flags = 144115188075921408, {{{lru = {next = 0x0 <fixed_percpu_data>,
          prev = 0xdead000000000122}, {__filler = 0x0 <fixed_percpu_data>,
          mlock_count = 290}, buddy_list = {next = 0x0 <fixed_percpu_data>,
          prev = 0xdead000000000122}, pcp_list = {next = 0x0 <fixed_percpu_data>,
          prev = 0xdead000000000122}}, mapping = 0x0 <fixed_percpu_data>, {index = 0,
        share = 0}, private = 0}, {pp_magic = 0, pp = 0xdead000000000122,
      _pp_mapping_pad = 0, dma_addr = 0, {dma_addr_upper = 0, pp_frag_count = {
          counter = 0}}}, {compound_head = 0}, {_pt_pad_1 = 0,
      pmd_huge_pte = 0xdead000000000122, _pt_pad_2 = 0, {pt_mm = 0x0 <fixed_percpu_data>,
        pt_frag_refcount = {counter = 0}}, ptl = 0x0 <fixed_percpu_data>}, {
      pgmap = 0x0 <fixed_percpu_data>, zone_device_data = 0xdead000000000122},
    callback_head = {next = 0x0 <fixed_percpu_data>, func = 0xdead000000000122}}, {
    _mapcount = {counter = -1}, page_type = 4294967295}, _refcount = {counter = 1}}
(gdb) p/x *$7->frags[0].bv_page
$20 = {flags = 0x200000000010000, {{{lru = {next = 0x0, prev = 0xdead000000000122}, {
          __filler = 0x0, mlock_count = 0x122}, buddy_list = {next = 0x0,
          prev = 0xdead000000000122}, pcp_list = {next = 0x0, prev = 0xdead000000000122}},
      mapping = 0x0, {index = 0x0, share = 0x0}, private = 0x0}, {pp_magic = 0x0,
      pp = 0xdead000000000122, _pp_mapping_pad = 0x0, dma_addr = 0x0, {dma_addr_upper = 0x0,
        pp_frag_count = {counter = 0x0}}}, {compound_head = 0x0}, {_pt_pad_1 = 0x0,
      pmd_huge_pte = 0xdead000000000122, _pt_pad_2 = 0x0, {pt_mm = 0x0, pt_frag_refcount = {
          counter = 0x0}}, ptl = 0x0}, {pgmap = 0x0, zone_device_data = 0xdead000000000122},
    callback_head = {next = 0x0, func = 0xdead000000000122}}, {_mapcount = {
      counter = 0xffffffff}, page_type = 0xffffffff}, _refcount = {counter = 0x1}}

(gdb) p *$7->frags[1].bv_page
$19 = {flags = 144115188075921408, {{{lru = {next = 0x0 <fixed_percpu_data>,
          prev = 0xdead000000000122}, {__filler = 0x0 <fixed_percpu_data>,
          mlock_count = 290}, buddy_list = {next = 0x0 <fixed_percpu_data>,
          prev = 0xdead000000000122}, pcp_list = {next = 0x0 <fixed_percpu_data>,
          prev = 0xdead000000000122}}, mapping = 0x0 <fixed_percpu_data>, {index = 0,
        share = 0}, private = 0}, {pp_magic = 0, pp = 0xdead000000000122,
      _pp_mapping_pad = 0, dma_addr = 0, {dma_addr_upper = 0, pp_frag_count = {
          counter = 0}}}, {compound_head = 0}, {_pt_pad_1 = 0,
      pmd_huge_pte = 0xdead000000000122, _pt_pad_2 = 0, {pt_mm = 0x0 <fixed_percpu_data>,
        pt_frag_refcount = {counter = 0}}, ptl = 0x0 <fixed_percpu_data>}, {
      pgmap = 0x0 <fixed_percpu_data>, zone_device_data = 0xdead000000000122},
    callback_head = {next = 0x0 <fixed_percpu_data>, func = 0xdead000000000122}}, {
    _mapcount = {counter = -1}, page_type = 4294967295}, _refcount = {counter = 4}}
(gdb) p/x *$7->frags[1].bv_page
$21 = {flags = 0x200000000010000, {{{lru = {next = 0x0, prev = 0xdead000000000122}, {
          __filler = 0x0, mlock_count = 0x122}, buddy_list = {next = 0x0,
          prev = 0xdead000000000122}, pcp_list = {next = 0x0, prev = 0xdead000000000122}},
      mapping = 0x0, {index = 0x0, share = 0x0}, private = 0x0}, {pp_magic = 0x0,
      pp = 0xdead000000000122, _pp_mapping_pad = 0x0, dma_addr = 0x0, {dma_addr_upper = 0x0,
        pp_frag_count = {counter = 0x0}}}, {compound_head = 0x0}, {_pt_pad_1 = 0x0,
      pmd_huge_pte = 0xdead000000000122, _pt_pad_2 = 0x0, {pt_mm = 0x0, pt_frag_refcount = {
          counter = 0x0}}, ptl = 0x0}, {pgmap = 0x0, zone_device_data = 0xdead000000000122},
    callback_head = {next = 0x0, func = 0xdead000000000122}}, {_mapcount = {
      counter = 0xffffffff}, page_type = 0xffffffff}, _refcount = {counter = 0x4}}

** Sandbox
*** CONFIG options for tracing

CONFIG_DEBUG_INFO=y
CONFIG_FRAME_POINTER=y
CONFIG_FTRACE=y
CONFIG_KALLSYMS=y
CONFIG_KPROBES=y
CONFIG_KPROBE_EVENTS=y
CONFIG_LOCKDEP=y
CONFIG_LOCK_STAT=y
CONFIG_PERF_EVENTS=y
CONFIG_TRACEPOINTS=y
CONFIG_UPROBES=y
CONFIG_UPROBE_EVENTS=y

*** Sample PPS and packet send times for 1500B IP packets

#+begin_src C :includes <stdio.h> :includes <stdint.h>
#include <stdio.h>
#define ENET_OHEAD (14 + 4 + 8 + 12)
#define _1GE_PPS(iptfs_ip_mtu) ((1e9 / 8) / ((iptfs_ip_mtu) + ENET_OHEAD))
#define _10GE_PPS(iptfs_ip_mtu) ((1e10 / 8) / ((iptfs_ip_mtu) + ENET_OHEAD))
#define _40GE_PPS(iptfs_ip_mtu) ((4e10 / 8) / ((iptfs_ip_mtu) + ENET_OHEAD))
#define _100GE_PPS(iptfs_ip_mtu) ((1e11 / 8) / ((iptfs_ip_mtu) + ENET_OHEAD))
#define _1GE_PP_NANOS(iptfs_ip_mtu) (1e9 / _1GE_PPS(iptfs_ip_mtu))
#define _10GE_PP_NANOS(iptfs_ip_mtu) (1e9 / _10GE_PPS(iptfs_ip_mtu))
#define _40GE_PP_NANOS(iptfs_ip_mtu) (1e9 / _40GE_PPS(iptfs_ip_mtu))
#define _100GE_PP_NANOS(iptfs_ip_mtu) (1e9 / _100GE_PPS(iptfs_ip_mtu))

int mtu = 64;

printf("+ 1GE 10GE 40GE 100GE\n");
printf("PPS %lu %lu %lu %lu\n", (uint64_t)_1GE_PPS(mtu),(uint64_t)_10GE_PPS(mtu),(uint64_t)_40GE_PPS(mtu),(uint64_t)_100GE_PPS(mtu));
printf("packet-time %luns %luns %luns %luns\n", (uint64_t)_1GE_PP_NANOS(mtu),(uint64_t)_10GE_PP_NANOS(mtu),(uint64_t)_40GE_PP_NANOS(mtu),(uint64_t)_100GE_PP_NANOS(mtu));
#+end_src

#+RESULTS:
| +           | 1GE     | 10GE     | 40GE     | 100GE     |
| PPS         | 1225490 | 12254901 | 49019607 | 122549019 |
| packet-time | 816ns   | 81ns     | 20ns     | 8ns       |

*** Checking uncommon flag setting code disassembly

No idea how to get this emacs-babel to work, but this saves the code
for now. Babel is always evaluating the C block and saving those
empty results in the file not the `code` as instructed.

#+NAME: disassemble
#+BEGIN_SRC bash :var filename=check-c-flag-opt :results output
  echo "$filename"
  gcc -O2 -c -o ${filename%.c}.o $filename
  echo objdump -S -d "${filename%.c}.o"
#+END_SRC

# #+header: :post disassemble
#+header: :exports code :results output file :file-ext c :output-dir /tmp
#+begin_src C
  #define SETF (1<<12)
  #define CHECKF (1ull << 38)

  unsigned long long is_setdm(unsigned long long bits, unsigned long long result)
  {
      result |= (bits & CHECKF) / CHECKF * SETF;
      return result;
  }

  unsigned long long is_setto(unsigned long long bits, unsigned long long result)
  {
      result |= bits & CHECKF ? SETF : 0;
      return result;
  }

  unsigned long long is_setif(unsigned long long bits, unsigned long long result)
  {
      if (!!(bits & CHECKF))
          result |= SETF;
      return result;
  }
#+end_src

#+begin_src bash
  echo "Hello World"
#+end_src

*** Performance Triaging

Testing done on a single server with 3 networks cards using munet and wiring the
ports to each other.
[h1] - [r1] - [r2] - [h2]
       ===========
**** Qemu emulated - single socket/core

- Routed from h1 to h2 iperf bidir:              ~2000 Mbps
- IPsec [r1,r2] tunnel, from h1 to h2 iperf bidir ~120 Mbps
- IPTFS [r1,r2] tunnel, from h1 to h2 iperf bidir   ~2 Mbps

**** Qemu -accel kvm - single socket/core

- Routed from h1 to h2 iperf bidir:              ~9400 Mbps
- IPsec [r1,r2] tunnel, from h1 to h2 iperf bidir ~920 Mbps
- IPTFS [r1,r2] tunnel, from h1 to h2 iperf bidir   ~2 Mbps

**** Qemu -accel kvm - 4 sockets

- Routed from h1 to h2 iperf bidir:               ~9400 Mbps
- IPsec [r1,r2] tunnel, from h1 to h2 iperf bidir ~7200 Mbps
- IPTFS [r1,r2] tunnel, from h1 to h2 iperf bidir     700Kbps-3.87Mbps

** Examples
*** These are single Core

**** tests/stress/trex_stress_phy.py::test_policy_imix

***** mode == tunnel (i.e., normal ipsec) (50%/50%)

Global Statistics

connection   : localhost, Port 4501                       total_tx_L2  : 3.83 Gbps
version      : STL @ v2.98                                total_tx_L1  : 3.93 Gbps
cpu_util.    : 6.52% @ 2 cores (2 per dual port)          total_rx     : 3.04 Gbps
rx_cpu_util. : 8.26% / 0 pps                              total_pps    : 645.97 Kpps
async_util.  : 0% / 30.41 bps                             drop_rate    : 792.36 Mbps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |             6.52% |             6.52% |
--         |                   |                   |
Tx bps L2  |         1.92 Gbps |         1.91 Gbps |         3.83 Gbps
Tx bps L1  |         1.97 Gbps |         1.96 Gbps |         3.93 Gbps
Tx pps     |       323.81 Kpps |       322.16 Kpps |       645.97 Kpps
Line Util. |            4.93 % |             4.9 % |
---        |                   |                   |
Rx bps     |         1.52 Gbps |         1.51 Gbps |         3.04 Gbps
Rx pps     |       256.81 Kpps |        255.5 Kpps |       512.31 Kpps
----       |                   |                   |
opackets   |           7479318 |           7511222 |          14990540
ipackets   |           5941015 |           5965888 |          11906903
obytes     |        5542174638 |        5565815502 |       11107990140
ibytes     |        4402286022 |        4421020888 |        8823306910
tx-pkts    |        7.48 Mpkts |        7.51 Mpkts |       14.99 Mpkts
rx-pkts    |        5.94 Mpkts |        5.97 Mpkts |       11.91 Mpkts
tx-bytes   |           5.54 GB |           5.57 GB |          11.11 GB
rx-bytes   |            4.4 GB |           4.42 GB |           8.82 GB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

status:  /

browse:     'q' - quit, 'd' - dashboard, 'u' - util, 's' - streams, 'l' - latency,
dashboard:  'n' - reset view, 'o' - owned ports, 'a' - all ports, 'c' - clear,

***** mode == iptfs (imix new 50%/50%)

Global Statistics

connection   : localhost, Port 4501                       total_tx_L2  : 3.84 Gbps
version      : STL @ v2.98                                total_tx_L1  : 3.94 Gbps
cpu_util.    : 6.39% @ 2 cores (2 per dual port)          total_rx     : 2.9 Gbps
rx_cpu_util. : 6.74% / 0 pps                              total_pps    : 647.42 Kpps
async_util.  : 0% / 34.3 bps                              drop_rate    : 942.81 Mbps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |             6.39% |             6.39% |
--         |                   |                   |
Tx bps L2  |         1.92 Gbps |         1.92 Gbps |         3.84 Gbps
Tx bps L1  |         1.97 Gbps |         1.97 Gbps |         3.94 Gbps
Tx pps     |       323.87 Kpps |       324.12 Kpps |       647.99 Kpps
Line Util. |            4.93 % |            4.93 % |
---        |                   |                   |
Rx bps     |         1.45 Gbps |         1.45 Gbps |          2.9 Gbps
Rx pps     |       244.29 Kpps |       244.59 Kpps |       488.88 Kpps
----       |                   |                   |
opackets   |           6113860 |           6158922 |          12272782
ipackets   |           4643520 |           4678016 |           9321536
obytes     |        4530370260 |        4563761202 |        9094131462
ibytes     |        3440826656 |        3466377360 |        6907204016
tx-pkts    |        6.11 Mpkts |        6.16 Mpkts |       12.27 Mpkts
rx-pkts    |        4.64 Mpkts |        4.68 Mpkts |        9.32 Mpkts
tx-bytes   |           4.53 GB |           4.56 GB |           9.09 GB
rx-bytes   |           3.44 GB |           3.47 GB |           6.91 GB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

status:  /

browse:     'q' - quit, 'd' - dashboard, 'u' - util, 's' - streams, 'l' - latency,
dashboard:  'n' - reset view, 'o' - owned ports, 'a' - all ports, 'c' - clear,

***** mode == iptfs (imix legacy/firewall 7,4,1)

Global Statistics

connection   : localhost, Port 4501                       total_tx_L2  : 3.98 Gbps
version      : STL @ v2.98                                total_tx_L1  : 4.2 Gbps
cpu_util.    : 12.07% @ 2 cores (2 per dual port)         total_rx     : 2.02 Gbps
rx_cpu_util. : 9.33% / 0 pps                              total_pps    : 1.41 Mpps
async_util.  : 0% / 36.53 bps                             drop_rate    : 1.95 Gbps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |            12.07% |            12.07% |
--         |                   |                   |
Tx bps L2  |            2 Gbps |         1.98 Gbps |         3.98 Gbps
Tx bps L1  |         2.11 Gbps |         2.09 Gbps |          4.2 Gbps
Tx pps     |       706.35 Kpps |       700.51 Kpps |         1.41 Mpps
Line Util. |            5.28 % |            5.23 % |
---        |                   |                   |
Rx bps     |         1.02 Gbps |         1.01 Gbps |         2.02 Gbps
Rx pps     |       359.91 Kpps |       356.86 Kpps |       716.76 Kpps
----       |                   |                   |
opackets   |          10937602 |          10984861 |          21922463
ipackets   |           5620736 |           5645504 |          11266240
obytes     |        3866441526 |        3883148074 |        7749589600
ibytes     |        1985575700 |        1995323444 |        3980899144
tx-pkts    |       10.94 Mpkts |       10.98 Mpkts |       21.92 Mpkts
rx-pkts    |        5.62 Mpkts |        5.65 Mpkts |       11.27 Mpkts
tx-bytes   |           3.87 GB |           3.88 GB |           7.75 GB
rx-bytes   |           1.99 GB |              2 GB |           3.98 GB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

status:  \

browse:     'q' - quit, 'd' - dashboard, 'u' - util, 's' - streams, 'l' - latency,
dashboard:  'n' - reset view, 'o' - owned ports, 'a' - all ports, 'c' - clear,

***** mode == tunnel (i.e., normal ipsec) (imix legacy 7,4,1)

Global Statistics

connection   : localhost, Port 4501                       total_tx_L2  : 3.94 Gbps
version      : STL @ v2.98                                total_tx_L1  : 4.16 Gbps
cpu_util.    : 11.57% @ 2 cores (2 per dual port)         total_rx     : 1.5 Gbps
rx_cpu_util. : 8.49% / 0 pps                              total_pps    : 1.39 Mpps
async_util.  : 0% / 0 bps                                 drop_rate    : 2.43 Gbps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |            11.57% |            11.57% |
--         |                   |                   |
Tx bps L2  |         1.97 Gbps |         1.97 Gbps |         3.94 Gbps
Tx bps L1  |         2.08 Gbps |         2.08 Gbps |         4.16 Gbps
Tx pps     |       694.97 Kpps |       697.46 Kpps |         1.39 Mpps
Line Util. |            5.19 % |            5.21 % |
---        |                   |                   |
Rx bps     |       750.55 Mbps |       754.34 Mbps |          1.5 Gbps
Rx pps     |       265.62 Kpps |       266.53 Kpps |       532.14 Kpps
----       |                   |                   |
opackets   |         313742270 |         313798138 |         627540408
ipackets   |         120083200 |         120102542 |         240185742
obytes     |      110907888436 |      110927643186 |      221835531622
ibytes     |       42428351448 |       42428427258 |       84856778706
tx-pkts    |      313.74 Mpkts |       313.8 Mpkts |      627.54 Mpkts
rx-pkts    |      120.08 Mpkts |       120.1 Mpkts |      240.19 Mpkts
tx-bytes   |         110.91 GB |         110.93 GB |         221.84 GB
rx-bytes   |          42.43 GB |          42.43 GB |          84.86 GB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

status:  \

browse:     'q' - quit, 'd' - dashboard, 'u' - util, 's' - streams, 'l' - latency,
dashboard:  'n' - reset view, 'o' - owned ports, 'a' - all ports, 'c' - clear,

*** These are 3 core

NOTE: for Qemu if multiple cores are used ipsec/iptfs seems to only use 1 thread
(core). If multple *sockets* are used then multiple threads (sockets) are utilized.

**** tests/stress/trex_stress_phy.py::test_policy_imix

***** New IMIX (50/50)

****** mode == tunnel (i.e., normal ipsec) (50%/50%)

===== 0 DROP May go faster =====

Global Statistics

connection   : localhost, Port 4501                       total_tx_L2  : 3.87 Gbps
version      : STL @ v2.98                                total_tx_L1  : 3.98 Gbps
cpu_util.    : 7.67% @ 2 cores (2 per dual port)          total_rx     : 3.87 Gbps
rx_cpu_util. : 10.18% / 0.12 pps                          total_pps    : 653.14 Kpps
async_util.  : 0% / 0 bps                                 drop_rate    : 0 bps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |             7.67% |             7.67% |
--         |                   |                   |
Tx bps L2  |         1.95 Gbps |         1.92 Gbps |         3.87 Gbps
Tx bps L1  |            2 Gbps |         1.97 Gbps |         3.98 Gbps
Tx pps     |       328.91 Kpps |       324.28 Kpps |       653.19 Kpps
Line Util. |            5.01 % |            4.94 % |
---        |                   |                   |
Rx bps     |         1.95 Gbps |         1.92 Gbps |         3.87 Gbps
Rx pps     |       328.91 Kpps |       324.25 Kpps |       653.16 Kpps
----       |                   |                   |
opackets   |          37006482 |          37035660 |          74042142
ipackets   |          37006464 |          37035618 |          74042082
obytes     |       27421803162 |       27443424060 |       54865227222
ibytes     |       27421789824 |       27443391520 |       54865181344
tx-pkts    |       37.01 Mpkts |       37.04 Mpkts |       74.04 Mpkts
rx-pkts    |       37.01 Mpkts |       37.04 Mpkts |       74.04 Mpkts
tx-bytes   |          27.42 GB |          27.44 GB |          54.87 GB
rx-bytes   |          27.42 GB |          27.44 GB |          54.87 GB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

status:  /

browse:     'q' - quit, 'd' - dashboard, 'u' - util, 's' - streams, 'l' - latency,
dashboard:  'n' - reset view, 'o' - owned ports, 'a' - all ports, 'c' - clear,

****** mode == iptfs (imix new 50%/50%)

Global Statistics

connection   : localhost, Port 4501                       total_tx_L2  : 3.85 Gbps
version      : STL @ v2.98                                total_tx_L1  : 3.95 Gbps
cpu_util.    : 6.35% @ 2 cores (2 per dual port)          total_rx     : 2.7 Gbps
rx_cpu_util. : 6.64% / 0 pps                              total_pps    : 649 Kpps
async_util.  : 0% / 33.17 bps                             drop_rate    : 1.14 Gbps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |             6.35% |             6.35% |
--         |                   |                   |
Tx bps L2  |         1.93 Gbps |         1.93 Gbps |         3.86 Gbps
Tx bps L1  |         1.98 Gbps |         1.99 Gbps |         3.97 Gbps
Tx pps     |       325.41 Kpps |       326.34 Kpps |       651.75 Kpps
Line Util. |            4.95 % |            4.97 % |
---        |                   |                   |
Rx bps     |         1.35 Gbps |         1.36 Gbps |         2.71 Gbps
Rx pps     |       228.17 Kpps |       228.88 Kpps |       457.05 Kpps
----       |                   |                   |
opackets   |           7075748 |           7094883 |          14170631
ipackets   |           5048896 |           5062638 |          10111534
obytes     |        5243129268 |        5257307626 |       10500436894
ibytes     |        3741435036 |        3751414630 |        7492849666
tx-pkts    |        7.08 Mpkts |        7.09 Mpkts |       14.17 Mpkts
rx-pkts    |        5.05 Mpkts |        5.06 Mpkts |       10.11 Mpkts
tx-bytes   |           5.24 GB |           5.26 GB |           10.5 GB
rx-bytes   |           3.74 GB |           3.75 GB |           7.49 GB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

status:  \

browse:     'q' - quit, 'd' - dashboard, 'u' - util, 's' - streams, 'l' - latency,
dashboard:  'n' - reset view, 'o' - owned ports, 'a' - all ports, 'c' - clear,

***** Old Imix 7,4,1

****** mode == iptfs (imix legacy/firewall 7,4,1)

Global Statistics

connection   : localhost, Port 4501                       total_tx_L2  : 3.93 Gbps
version      : STL @ v2.98                                total_tx_L1  : 4.16 Gbps
cpu_util.    : 14.76% @ 2 cores (2 per dual port)         total_rx     : 3.56 Gbps
rx_cpu_util. : 18.21% / 0 pps                             total_pps    : 1.39 Mpps
async_util.  : 0% / 30.14 bps                             drop_rate    : 0 bps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |            14.76% |            14.76% |
--         |                   |                   |
Tx bps L2  |         1.97 Gbps |         1.97 Gbps |         3.94 Gbps
Tx bps L1  |         2.08 Gbps |         2.09 Gbps |         4.17 Gbps
Tx pps     |       696.39 Kpps |       697.98 Kpps |         1.39 Mpps
Line Util. |             5.2 % |            5.21 % |
---        |                   |                   |
Rx bps     |         1.86 Gbps |         1.71 Gbps |         3.58 Gbps
Rx pps     |       658.15 Kpps |       606.26 Kpps |         1.26 Mpps
----       |                   |                   |
opackets   |          14995714 |          15099834 |          30095548
ipackets   |          14231296 |          13124265 |          27355561
obytes     |        5300982114 |        5337789222 |       10638771336
ibytes     |        5030619874 |        4638994972 |        9669614846
tx-pkts    |          15 Mpkts |        15.1 Mpkts |        30.1 Mpkts
rx-pkts    |       14.23 Mpkts |       13.12 Mpkts |       27.36 Mpkts
tx-bytes   |            5.3 GB |           5.34 GB |          10.64 GB
rx-bytes   |           5.03 GB |           4.64 GB |           9.67 GB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

status:  \

browse:     'q' - quit, 'd' - dashboard, 'u' - util, 's' - streams, 'l' - latency,
dashboard:  'n' - reset view, 'o' - owned ports, 'a' - all ports, 'c' - clear,

****** mode == tunnel (i.e., normal ipsec) (imix legacy 7,4,1)

Global Statistics

connection   : localhost, Port 4501                       total_tx_L2  : 3.95 Gbps
version      : STL @ v2.98                                total_tx_L1  : 4.17 Gbps
cpu_util.    : 10.12% @ 2 cores (2 per dual port)         total_rx     : 1.41 Gbps
rx_cpu_util. : 5.47% / 0 pps                              total_pps    : 1.4 Mpps
async_util.  : 0% / 34.58 bps                             drop_rate    : 2.54 Gbps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |            10.12% |            10.12% |
--         |                   |                   |
Tx bps L2  |         1.97 Gbps |         1.98 Gbps |         3.95 Gbps
Tx bps L1  |         2.08 Gbps |         2.09 Gbps |         4.17 Gbps
Tx pps     |       697.28 Kpps |       698.79 Kpps |          1.4 Mpps
Line Util. |            5.21 % |            5.22 % |
---        |                   |                   |
Rx bps     |       704.24 Mbps |       703.04 Mbps |         1.41 Gbps
Rx pps     |       247.81 Kpps |       248.36 Kpps |       496.17 Kpps
----       |                   |                   |
opackets   |          11321860 |          11419036 |          22740896
ipackets   |           4065024 |           4100397 |           8165421
obytes     |        4002278236 |        4036629952 |        8038908188
ibytes     |        1453675184 |        1455668512 |        2909343696
tx-pkts    |       11.32 Mpkts |       11.42 Mpkts |       22.74 Mpkts
rx-pkts    |        4.07 Mpkts |         4.1 Mpkts |        8.17 Mpkts
tx-bytes   |              4 GB |           4.04 GB |           8.04 GB
rx-bytes   |           1.45 GB |           1.46 GB |           2.91 GB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

status:  \

browse:     'q' - quit, 'd' - dashboard, 'u' - util, 's' - streams, 'l' - latency,
dashboard:  'n' - reset view, 'o' - owned ports, 'a' - all ports, 'c' - clear,

****** mode == iptfs (i.e., normal ipsec) 1G 40 octet packet
****** mode == tunnel (i.e., normal ipsec) 1G 40 octet packet

- tunnel 1G 40b pkt
- Global Statistics

connection   : localhost, Port 4501                       total_tx_L2  : 3 Gbps
version      : STL @ v2.98                                total_tx_L1  : 3.94 Gbps
cpu_util.    : 36.57% @ 2 cores (2 per dual port)         total_rx     : 494 Mbps
rx_cpu_util. : 13.65% / 0 pps                             total_pps    : 5.86 Mpps
async_util.  : 0% / 35.1 bps                              drop_rate    : 2.51 Gbps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |            36.57% |            36.57% |
--         |                   |                   |
Tx bps L2  |          1.5 Gbps |          1.5 Gbps |            3 Gbps
Tx bps L1  |         1.97 Gbps |         1.97 Gbps |         3.94 Gbps
Tx pps     |         2.93 Mpps |         2.93 Mpps |         5.86 Mpps
Line Util. |            4.93 % |            4.92 % |
---        |                   |                   |
Rx bps     |       250.51 Mbps |       243.49 Mbps |          494 Mbps
Rx pps     |       489.27 Kpps |       475.57 Kpps |       964.84 Kpps
----       |                   |                   |
opackets   |          89007203 |          89443956 |         178451159
ipackets   |          14835660 |          14522048 |          29357708
obytes     |        5696460992 |        5724413184 |       11420874176
ibytes     |         949482112 |         929411072 |        1878893184
tx-pkts    |       89.01 Mpkts |       89.44 Mpkts |      178.45 Mpkts
rx-pkts    |       14.84 Mpkts |       14.52 Mpkts |       29.36 Mpkts
tx-bytes   |            5.7 GB |           5.72 GB |          11.42 GB
rx-bytes   |         949.48 MB |         929.41 MB |           1.88 GB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

status:  |

browse:     'q' - quit, 'd' - dashboard, 'u' - util, 's' - streams, 'l' - latency,
dashboard:  'n' - reset view, 'o' - owned ports, 'a' - all ports, 'c' - clear,

** More performance

*** routed, no tunnel

**** UPKT 1400, rate=8G, 4core

Global Statistics

connection   : 192.168.0.1, Port 4501                     total_tx_L2  : 15.17 Gbps
version      : STL @ v2.98                                total_tx_L1  : 15.39 Gbps
cpu_util.    : 4.14% @ 4 cores (4 per dual port)          total_rx     : 14.14 Gbps
rx_cpu_util. : 19.92% / 0 pps                             total_pps    : 1.34 Mpps
async_util.  : 0% / 0 bps                                 drop_rate    : 0 bps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |             4.14% |             4.14% |
--         |                   |                   |
Tx bps L2  |         7.57 Gbps |         7.61 Gbps |        15.18 Gbps
Tx bps L1  |         7.68 Gbps |         7.71 Gbps |        15.39 Gbps
Tx pps     |       667.53 Kpps |       670.59 Kpps |         1.34 Mpps
Line Util. |            19.2 % |           19.29 % |
---        |                   |                   |
Rx bps     |         6.55 Gbps |          7.6 Gbps |        14.15 Gbps
Rx pps     |       577.24 Kpps |       670.25 Kpps |         1.25 Mpps
----       |                   |                   |
opackets   |          16176393 |          16229842 |          32406235
ipackets   |          14131488 |          16224312 |          30355800
obytes     |       22938123860 |       23013915956 |       45952039816
ibytes     |       20038449984 |       23006072998 |       43044522982
tx-pkts    |       16.18 Mpkts |       16.23 Mpkts |       32.41 Mpkts
rx-pkts    |       14.13 Mpkts |       16.22 Mpkts |       30.36 Mpkts
tx-bytes   |          22.94 GB |          23.01 GB |          45.95 GB
rx-bytes   |          20.04 GB |          23.01 GB |          43.04 GB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

status:  \

Press 'ESC' for navigation panel...
status:

tui(read-only)>

**** UPKT 1K, rate=8G, 4core

Global Statistics

connection   : 192.168.0.1, Port 4501                     total_tx_L2  : 15.35 Gbps
version      : STL @ v2.98                                total_tx_L1  : 15.66 Gbps
cpu_util.    : 4.76% @ 4 cores (4 per dual port)          total_rx     : 11.28 Gbps
rx_cpu_util. : 20.4% / 0 pps                              total_pps    : 1.89 Mpps
async_util.  : 0% / 0 bps                                 drop_rate    : 4.08 Gbps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |             4.76% |             4.76% |
--         |                   |                   |
Tx bps L2  |          7.7 Gbps |         7.65 Gbps |        15.35 Gbps
Tx bps L1  |         7.85 Gbps |          7.8 Gbps |        15.66 Gbps
Tx pps     |       945.93 Kpps |       939.49 Kpps |         1.89 Mpps
Line Util. |           19.64 % |            19.5 % |
---        |                   |                   |
Rx bps     |         6.09 Gbps |         5.19 Gbps |        11.28 Gbps
Rx pps     |       747.53 Kpps |       637.13 Kpps |         1.38 Mpps
----       |                   |                   |
opackets   |          39428467 |          39494064 |          78922531
ipackets   |          31190218 |          26685961 |          57876179
obytes     |       40138179406 |       40204954110 |       80343133516
ibytes     |       31751641924 |       27166307280 |       58917949204
tx-pkts    |       39.43 Mpkts |       39.49 Mpkts |       78.92 Mpkts
rx-pkts    |       31.19 Mpkts |       26.69 Mpkts |       57.88 Mpkts
tx-bytes   |          40.14 GB |           40.2 GB |          80.34 GB
rx-bytes   |          31.75 GB |          27.17 GB |          58.92 GB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

status:  /

Press 'ESC' for navigation panel...
status:

tui(read-only)>

**** UPKT 512 rate=8G, 4core

Global Statistics

connection   : 192.168.0.1, Port 4501                     total_tx_L2  : 15.51 Gbps
version      : STL @ v2.98                                total_tx_L1  : 16.1 Gbps
cpu_util.    : 4.93% @ 4 cores (4 per dual port)          total_rx     : 3.21 Gbps
rx_cpu_util. : 10.02% / 0 pps                             total_pps    : 3.66 Mpps
async_util.  : 0% / 0 bps                                 drop_rate    : 12.3 Gbps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |             4.93% |             4.93% |
--         |                   |                   |
Tx bps L2  |         7.79 Gbps |         7.72 Gbps |        15.51 Gbps
Tx bps L1  |         8.09 Gbps |         8.01 Gbps |         16.1 Gbps
Tx pps     |         1.84 Mpps |         1.82 Mpps |         3.66 Mpps
Line Util. |           20.22 % |           20.03 % |
---        |                   |                   |
Rx bps     |         1.61 Gbps |          1.6 Gbps |         3.21 Gbps
Rx pps     |       380.82 Kpps |       377.25 Kpps |       758.07 Kpps
----       |                   |                   |
opackets   |          63793853 |          63793814 |         127587667
ipackets   |          13199104 |          13199116 |          26398220
obytes     |       33810742616 |       33810724050 |       67621466666
ibytes     |        6995525120 |        6995532010 |       13991057130
tx-pkts    |       63.79 Mpkts |       63.79 Mpkts |      127.59 Mpkts
rx-pkts    |        13.2 Mpkts |        13.2 Mpkts |        26.4 Mpkts
tx-bytes   |          33.81 GB |          33.81 GB |          67.62 GB
rx-bytes   |              7 GB |              7 GB |          13.99 GB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

status:  -

Press 'ESC' for navigation panel...
status:

tui(read-only)>

**** UPKT 256 rate=8G, 4core

Global Statistics

connection   : 192.168.0.1, Port 4501                     total_tx_L2  : 16.08 Gbps
version      : STL @ v2.98                                total_tx_L1  : 17.26 Gbps
cpu_util.    : 7.38% @ 4 cores (4 per dual port)          total_rx     : 1.66 Gbps
rx_cpu_util. : 10.18% / 0 pps                             total_pps    : 7.34 Mpps
async_util.  : 0% / 0 bps                                 drop_rate    : 14.42 Gbps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |             7.38% |             7.38% |
--         |                   |                   |
Tx bps L2  |         8.04 Gbps |         8.04 Gbps |        16.08 Gbps
Tx bps L1  |         8.63 Gbps |         8.63 Gbps |        17.26 Gbps
Tx pps     |         3.67 Mpps |         3.67 Mpps |         7.34 Mpps
Line Util. |           21.57 % |           21.57 % |
---        |                   |                   |
Rx bps     |       830.78 Mbps |       830.68 Mbps |         1.66 Gbps
Rx pps     |       379.01 Kpps |       378.96 Kpps |       757.97 Kpps
----       |                   |                   |
opackets   |         105193737 |         105523395 |         210717132
ipackets   |          10867507 |          10901568 |          21769075
obytes     |       28823082318 |       28913409150 |       57736491468
ibytes     |        2977696918 |        2987029632 |        5964726550
tx-pkts    |      105.19 Mpkts |      105.52 Mpkts |      210.72 Mpkts
rx-pkts    |       10.87 Mpkts |        10.9 Mpkts |       21.77 Mpkts
tx-bytes   |          28.82 GB |          28.91 GB |          57.74 GB
rx-bytes   |           2.98 GB |           2.99 GB |           5.96 GB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

status:  -

Press 'ESC' for navigation panel...
status:

tui(read-only)>

*** iptfs

**** 1400UPKT 5G rate 4core

Global Statistics

connection   : 192.168.0.1, Port 4501                     total_tx_L2  : 9.47 Gbps
version      : STL @ v2.98                                total_tx_L1  : 9.61 Gbps
cpu_util.    : 1.25% @ 4 cores (4 per dual port)          total_rx     : 2.86 Gbps
rx_cpu_util. : 3.06% / 0 pps                              total_pps    : 835.12 Kpps
async_util.  : 0% / 7.62 bps                              drop_rate    : 6.62 Gbps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |             1.25% |             1.25% |
--         |                   |                   |
Tx bps L2  |         4.72 Gbps |         4.75 Gbps |         9.47 Gbps
Tx bps L1  |         4.79 Gbps |         4.82 Gbps |          9.6 Gbps
Tx pps     |       416.15 Kpps |       418.74 Kpps |       834.89 Kpps
Line Util. |           11.97 % |           12.04 % |
---        |                   |                   |
Rx bps     |          1.4 Gbps |         1.46 Gbps |         2.85 Gbps
Rx pps     |        123.1 Kpps |       128.47 Kpps |       251.58 Kpps
----       |                   |                   |
opackets   |          17519987 |          17561715 |          35081702
ipackets   |           5292758 |           5474423 |          10767181
obytes     |       24843341566 |       24902511870 |       49745853436
ibytes     |        7505130844 |        7762730396 |       15267861240
tx-pkts    |       17.52 Mpkts |       17.56 Mpkts |       35.08 Mpkts
rx-pkts    |        5.29 Mpkts |        5.47 Mpkts |       10.77 Mpkts
tx-bytes   |          24.84 GB |           24.9 GB |          49.75 GB
rx-bytes   |           7.51 GB |           7.76 GB |          15.27 GB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

status:  \

Press 'ESC' for navigation panel...
status:

tui(read-only)>

FAILED tests/stress/test_stress_phy.py::test_policy_small_pkt - Exception: FAILED: p0missed: 18908723 (75.29220704883792%) p1missed: 18699872 (74.46058807941534%)

*** ipsec

**** 1400UPKT 5G rate 4core

Global Statistics

connection   : 192.168.0.1, Port 4501                     total_tx_L2  : 9.48 Gbps
version      : STL @ v2.98                                total_tx_L1  : 9.61 Gbps
cpu_util.    : 1.17% @ 4 cores (4 per dual port)          total_rx     : 2.47 Gbps
rx_cpu_util. : 2.57% / 0 pps                              total_pps    : 835.27 Kpps
async_util.  : 0% / 0 bps                                 drop_rate    : 7.01 Gbps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |             1.17% |             1.17% |
--         |                   |                   |
Tx bps L2  |         4.75 Gbps |         4.74 Gbps |         9.49 Gbps
Tx bps L1  |         4.82 Gbps |          4.8 Gbps |         9.62 Gbps
Tx pps     |       418.75 Kpps |       417.45 Kpps |       836.21 Kpps
Line Util. |           12.04 % |           12.01 % |
---        |                   |                   |
Rx bps     |         1.24 Gbps |         1.23 Gbps |         2.47 Gbps
Rx pps     |       109.15 Kpps |        108.8 Kpps |       217.94 Kpps
----       |                   |                   |
opackets   |          10810806 |          10844294 |          21655100
ipackets   |           2844712 |           2853184 |           5697896
obytes     |       15329724322 |       15377208892 |       30706933214
ibytes     |        4033801616 |        4045814912 |        8079616528
tx-pkts    |       10.81 Mpkts |       10.84 Mpkts |       21.66 Mpkts
rx-pkts    |        2.84 Mpkts |        2.85 Mpkts |         5.7 Mpkts
tx-bytes   |          15.33 GB |          15.38 GB |          30.71 GB
rx-bytes   |           4.03 GB |           4.05 GB |           8.08 GB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

status:  |

Press 'ESC' for navigation panel...
status:

tui(read-only)>

FAILED tests/stress/test_stress_phy.py::test_policy_small_pkt - Exception: FAILED: p0missed: 18559218 (73.90052116795617%) p1missed: 18557335 (73.89302329378069%)

but then another run got:

and was showing more like 2G

FAILED tests/stress/test_stress_phy.py::test_policy_small_pkt - Exception: FAILED: p0missed: 14448450 (57.53194908692576%) p1missed: 14270021 (56.82146677611519%)

*** ipsec tunnel

Global Statistics

connection   : 192.168.0.1, Port 4501                     total_tx_L2  : 3 Gbps
version      : STL @ v2.98                                total_tx_L1  : 3.94 Gbps
cpu_util.    : 4.32% @ 4 cores (4 per dual port)          total_rx     : 147.22 Mbps
rx_cpu_util. : 3.51% / 0 pps                              total_pps    : 5.86 Mpps
async_util.  : 0% / 0 bps                                 drop_rate    : 2.85 Gbps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |             4.32% |             4.32% |
--         |                   |                   |
Tx bps L2  |          1.5 Gbps |          1.5 Gbps |            3 Gbps
Tx bps L1  |         1.97 Gbps |         1.97 Gbps |         3.94 Gbps
Tx pps     |         2.93 Mpps |         2.93 Mpps |         5.86 Mpps
Line Util. |            4.92 % |            4.92 % |
---        |                   |                   |
Rx bps     |        73.59 Mbps |        73.63 Mbps |       147.22 Mbps
Rx pps     |       143.74 Kpps |        143.8 Kpps |       287.54 Kpps
----       |                   |                   |
opackets   |         595000923 |         595235414 |        1190236337
ipackets   |          29586114 |          29596946 |          59183060
obytes     |       38080059072 |       38095066496 |       76175125568
ibytes     |        1893511296 |        1894204544 |        3787715840
tx-pkts    |         595 Mpkts |      595.24 Mpkts |        1.19 Gpkts
rx-pkts    |       29.59 Mpkts |        29.6 Mpkts |       59.18 Mpkts
tx-bytes   |          38.08 GB |           38.1 GB |          76.18 GB
rx-bytes   |           1.89 GB |           1.89 GB |           3.79 GB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

status:  |

Press 'ESC' for navigation panel...
status:

tui(read-only)>

**** R1 HTOP

    0[|||                                                                            2.7%] Tasks: 17, 0 thr, 67 kthr; 2 running
    1[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||100.0%] Load average: 0.99 0.59 0.25
    2[                                                                               0.0%] Uptime: 00:04:31
  Mem[|||||||                                                                 55.0M/3.83G]
  Swp[                                                                              0K/0K]

  [Main] [I/O]
  PID USER       PRI  NI  VIRT   RES   SHR S  CPU%-MEM%   TIME+  Command
  256 root        20   0  3164  2272  2116 S   0.7  0.1  0:00.02 /usr/sbin/dropbear -R -2
  260 root        20   0  4356  3132  2572 R   0.7  0.1  0:00.34 htop
    1 root        20   0  3432   648   576 S   0.0  0.0  0:00.15 init
  142 root        20   0  3432   520   460 S   0.0  0.0  0:00.00 /sbin/syslogd -n
  146 root        20   0  3432   492   428 S   0.0  0.0  0:00.00 /sbin/klogd -n
  171 root        20   0  3164  1768  1636 S   0.0  0.0  0:00.00 /usr/sbin/dropbear -R
  172 root        20   0  3432   512   444 S   0.0  0.0  0:00.00 /sbin/getty -L console 0 vt100
  173 root        20   0  4528  3620  3268 S   0.0  0.1  0:00.00 /bin/sh --
  175 root        20   0  3432   524   460 S   0.0  0.0  0:00.00 /sbin/getty -L ttyS0 115200 vt100
  176 root        20   0  4660  4020  3472 S   0.0  0.1  0:00.04 -bash
  177 root        20   0  4528  3604  3256 S   0.0  0.1  0:00.00 /bin/sh --
  178 root        20   0  4528  3656  3308 S   0.0  0.1  0:00.00 /bin/sh --
  254 root        20   0  3164  2276  2120 S   0.0  0.1  0:00.00 /usr/sbin/dropbear -R -2
  255 root        20   0 20904 14368  9300 S   0.0  0.4  0:00.21 perf record -F 997 -a -g -o /tmp/perf.data -- sleep 1200
  257 root        20   0  4660  3740  3396 S   0.0  0.1  0:00.00 bash
  258 root        20   0  3300   508   448 S   0.0  0.0  0:00.00 sleep 1200
  283 root        20   0  3432   524   456 S   0.0  0.0  0:00.00 /sbin/getty -n -l /bin/sh -L ttyS3 115200 vt100

*** IPTFS

Global Statistics

connection   : 192.168.0.1, Port 4501                     total_tx_L2  : 2.98 Gbps
version      : STL @ v2.98                                total_tx_L1  : 3.91 Gbps
cpu_util.    : 4.91% @ 4 cores (4 per dual port)          total_rx     : 275.2 Mbps
rx_cpu_util. : 7.96% / 0 pps                              total_pps    : 5.82 Mpps
async_util.  : 0% / 0 bps                                 drop_rate    : 2.7 Gbps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |             4.91% |             4.91% |
--         |                   |                   |
Tx bps L2  |         1.48 Gbps |          1.5 Gbps |         2.98 Gbps
Tx bps L1  |         1.94 Gbps |         1.97 Gbps |         3.91 Gbps
Tx pps     |         2.89 Mpps |         2.93 Mpps |         5.82 Mpps
Line Util. |            4.86 % |            4.91 % |
---        |                   |                   |
Rx bps     |       136.81 Mbps |       138.39 Mbps |        275.2 Mbps
Rx pps     |        267.2 Kpps |       270.29 Kpps |       537.49 Kpps
----       |                   |                   |
opackets   |          47519368 |          47490145 |          95009513
ipackets   |           4431611 |           4424832 |           8856443
obytes     |        3041239252 |        3039369040 |        6080608292
ibytes     |         283623104 |         283189248 |         566812352
tx-pkts    |       47.52 Mpkts |       47.49 Mpkts |       95.01 Mpkts
rx-pkts    |        4.43 Mpkts |        4.42 Mpkts |        8.86 Mpkts
tx-bytes   |           3.04 GB |           3.04 GB |           6.08 GB
rx-bytes   |         283.62 MB |         283.19 MB |         566.81 MB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |                 0 |                 0 |                 0

status:  \

Press 'ESC' for navigation panel...
status:

tui(read-only)>


    0[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||100.0%] Tasks: 17, 0 thr, 67 kthr; 2 running
    1[|||||||||||||||||||||||||||||||||||||||||||||||||||||||                       66.2%] Load average: 0.64 0.18 0.06
    2[|||||                                                                          5.4%] Uptime: 00:01:05
  Mem[|||||                                                                   54.2M/3.83G]
  Swp[                                                                              0K/0K]

  [Main] [I/O]
  PID USER       PRI  NI  VIRT   RES   SHR S  CPU%-MEM%   TIME+  Command
    1 root        20   0  3432   656   580 S   0.0  0.0  0:00.11 init
  142 root        20   0  3432   520   460 S   0.0  0.0  0:00.00 /sbin/syslogd -n
  146 root        20   0  3432   520   452 S   0.0  0.0  0:00.00 /sbin/klogd -n
  171 root        20   0  3164  1788  1656 S   0.0  0.0  0:00.00 /usr/sbin/dropbear -R
  172 root        20   0  3432   516   456 S   0.0  0.0  0:00.00 /sbin/getty -L console 0 vt100
  173 root        20   0  4528  3588  3236 S   0.0  0.1  0:00.00 /bin/sh --
  175 root        20   0  3432   516   452 S   0.0  0.0  0:00.00 /sbin/getty -L ttyS0 115200 vt100
  176 root        20   0  4660  3844  3292 S   0.0  0.1  0:00.03 -bash
  177 root        20   0  4528  3756  3408 S   0.0  0.1  0:00.00 /bin/sh --
  178 root        20   0  4528  3576  3216 S   0.0  0.1  0:00.00 /bin/sh --
  254 root        20   0  3164  2268  2112 S   0.0  0.1  0:00.00 /usr/sbin/dropbear -R -2
  255 root        20   0 20904 14340  9276 S   0.0  0.4  0:00.12 perf record -F 997 -a -g -o /tmp/perf.data -- sleep 1200
  256 root        20   0  3300   520   460 S   0.0  0.0  0:00.00 sleep 1200
  257 root        20   0  3164  2236  2080 S   0.0  0.1  0:00.00 /usr/sbin/dropbear -R -2
  258 root        20   0  4660  3676  3184 S   0.0  0.1  0:00.00 bash

** Poor performance 150Kpps @ 40 w 4 cores:

Global Statistics

connection   : 192.168.0.1, Port 4501                     total_tx_L2  : 3 Gbps
version      : STL @ v2.98                                total_tx_L1  : 3.94 Gbps
cpu_util.    : 13.12% @ 2 cores (2 per dual port)         total_rx     : 144.04 Mbps
rx_cpu_util. : 9.97% / 0 pps                              total_pps    : 5.87 Mpps
async_util.  : 0% / 23.93 bps                             drop_rate    : 2.86 Gbps
total_cps.   : 0 cps                                      queue_full   : 0 pkts

Port Statistics

   port    |         0         |         1         |       total
-----------+-------------------+-------------------+------------------
owner      |              root |              root |
link       |                UP |                UP |
state      |      TRANSMITTING |      TRANSMITTING |
speed      |           40 Gb/s |           40 Gb/s |
CPU util.  |            13.12% |            13.12% |
--         |                   |                   |
Tx bps L2  |          1.5 Gbps |          1.5 Gbps |            3 Gbps
Tx bps L1  |         1.97 Gbps |         1.97 Gbps |         3.94 Gbps
Tx pps     |         2.93 Mpps |         2.93 Mpps |         5.86 Mpps
Line Util. |            4.92 % |            4.93 % |
---        |                   |                   |
Rx bps     |        71.98 Mbps |        72.02 Mbps |          144 Mbps
Rx pps     |       140.58 Kpps |       140.66 Kpps |       281.24 Kpps
----       |                   |                   |
opackets   |        1380900371 |        1381337221 |        2762237592
ipackets   |          69166674 |          69183583 |         138350257
obytes     |       88377623744 |       88405582144 |      176783205888
ibytes     |        4426667136 |        4427749312 |        8854416448
tx-pkts    |        1.38 Gpkts |        1.38 Gpkts |        2.76 Gpkts
rx-pkts    |       69.17 Mpkts |       69.18 Mpkts |      138.35 Mpkts
tx-bytes   |          88.38 GB |          88.41 GB |         176.78 GB
rx-bytes   |           4.43 GB |           4.43 GB |           8.85 GB
-----      |                   |                   |
oerrors    |                 0 |                 0 |                 0
ierrors    |             1,052 |             1,116 |             2,168

status:  -

Press 'ESC' for navigation panel...
status:

tui(read-only)>

bash-5.1# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
  0:         14          0          0          0   IO-APIC   2-edge      timer
  1:          0          9          0          0   IO-APIC   1-edge      i8042
  3:        452          0          0          0   IO-APIC   3-edge      ttyS1
  4:          0          0         39          0   IO-APIC   4-edge      ttyS0
  8:          0          0          1          0   IO-APIC   8-edge      rtc0
  9:          0          0          0          0   IO-APIC   9-fasteoi   acpi
 12:        125          0          0          0   IO-APIC  12-edge      i8042
 16:          0          0          0          0   IO-APIC  16-fasteoi   i801_smbus
 24:          0          0          0          0   PCI-MSI 32768-edge      virtio0-config
 25:          0          0          0         26   PCI-MSI 32769-edge      virtio0-virtqueues
 26:         20          0          0          0   PCI-MSI 512000-edge      ahci[0000:00:1f.2]
 27:          0          0          0          0   PCI-MSI 49152-edge      virtio1-config
 28:          0          0        116          0   PCI-MSI 49153-edge      virtio1-input.0
 29:          0          0          0        126   PCI-MSI 49154-edge      virtio1-output.0
 30:          0         41          0          0   PCI-MSI 65536-edge      eth1-TxRx-0
 31:          0          0     147347          0   PCI-MSI 65537-edge      eth1-TxRx-1
 32:          0          0          0     147444   PCI-MSI 65538-edge      eth1-TxRx-2
 33:         39          0          0          0   PCI-MSI 65539-edge      eth1-TxRx-3
 34:          0          2          0          0   PCI-MSI 65540-edge      eth1
 35:          0          0          0          0   PCI-MSI 98304-edge      virtio2-config
 36:          0          3          0          0   PCI-MSI 98305-edge      virtio2-requests
 37:          0          0          0         42   PCI-MSI 81920-edge      iavf-0000:00:05.0:mbx
 38:          0          0          2          0   PCI-MSI 81921-edge      iavf-eth2-TxRx-0
 39:          0          0          0     333179   PCI-MSI 81922-edge      iavf-eth2-TxRx-1
 40:          0          0          0          0   PCI-MSI 81923-edge      iavf-eth2-TxRx-2
 41:          0          0          0          0   PCI-MSI 81924-edge      iavf-eth2-TxRx-3
NMI:          0          0          0          0   Non-maskable interrupts
LOC:      70266      70258     143816      70294   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0   Performance monitoring interrupts
IWI:          0          0          0          0   IRQ work interrupts
RTR:          0          0          0          0   APIC ICR read retries
RES:         13         20         21         18   Rescheduling interrupts
CAL:         72        103         73         88   Function call interrupts
TLB:          1          2          1          1   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0   Threshold APIC interrupts
DFR:          0          0          0          0   Deferred Error APIC interrupts
MCE:          0          0          0          0   Machine check exceptions
MCP:          1          1          1          1   Machine check polls
HYP:          1          1          1          1   Hypervisor callback interrupts
ERR:          0
MIS:          0
PIN:          0          0          0          0   Posted-interrupt notification event
NPI:          0          0          0          0   Nested posted-interrupt event
PIW:          0          0          0          0   Posted-interrupt wakeup event
bash-5.1#

bash-5.1# htop
bash-5.1# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
  0:         14          0          0          0   IO-APIC   2-edge      timer
  1:          0          9          0          0   IO-APIC   1-edge      i8042
  3:        606          0          0          0   IO-APIC   3-edge      ttyS1
  4:          0          0         49          0   IO-APIC   4-edge      ttyS0
  8:          0          0          1          0   IO-APIC   8-edge      rtc0
  9:          0          0          0          0   IO-APIC   9-fasteoi   acpi
 12:        125          0          0          0   IO-APIC  12-edge      i8042
 16:          0          0          0          0   IO-APIC  16-fasteoi   i801_smbus
 24:          0          0          0          0   PCI-MSI 32768-edge      virtio0-config
 25:          0          0          0         26   PCI-MSI 32769-edge      virtio0-virtqueues
 26:         29          0          0          0   PCI-MSI 512000-edge      ahci[0000:00:1f.2]
 27:          0          0          0          0   PCI-MSI 49152-edge      virtio1-config
 28:          0          0        167          0   PCI-MSI 49153-edge      virtio1-input.0
 29:          0          0          0        123   PCI-MSI 49154-edge      virtio1-output.0
 30:          0         46          0          0   PCI-MSI 65536-edge      eth1-TxRx-0
 31:          0          0         46          0   PCI-MSI 65537-edge      eth1-TxRx-1
 32:          0          0          0     161945   PCI-MSI 65538-edge      eth1-TxRx-2
 33:     193538          0          0          0   PCI-MSI 65539-edge      eth1-TxRx-3
 34:          0          3          0          0   PCI-MSI 65540-edge      eth1
 35:          0          0          0          0   PCI-MSI 98304-edge      virtio2-config
 36:          0          3          0          0   PCI-MSI 98305-edge      virtio2-requests
 37:          0          0          0         46   PCI-MSI 81920-edge      iavf-0000:00:05.0:mbx
 38:          0          0          2          0   PCI-MSI 81921-edge      iavf-eth2-TxRx-0
 39:          0          0          0          1   PCI-MSI 81922-edge      iavf-eth2-TxRx-1
 40:     364801          0          0          0   PCI-MSI 81923-edge      iavf-eth2-TxRx-2
 41:          0          0          0          0   PCI-MSI 81924-edge      iavf-eth2-TxRx-3
NMI:          0          0          0          0   Non-maskable interrupts
LOC:      76411      76355      76435     172972   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0   Performance monitoring interrupts
IWI:          0          0          0          0   IRQ work interrupts
RTR:          0          0          0          0   APIC ICR read retries
RES:         14          9         34         23   Rescheduling interrupts
CAL:         75        107         97         67   Function call interrupts
TLB:          2          2          1          2   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0   Threshold APIC interrupts
DFR:          0          0          0          0   Deferred Error APIC interrupts
MCE:          0          0          0          0   Machine check exceptions
MCP:          1          1          1          1   Machine check polls
HYP:          1          1          1          1   Hypervisor callback interrupts
ERR:          0
MIS:          0
PIN:          0          0          0          0   Posted-interrupt notification event
NPI:          0          0          0          0   Nested posted-interrupt event
PIW:          0          0          0          0   Posted-interrupt wakeup event
b
but even moving the IRQ didn't change anything maybe b/c rx-2 is being shared?

* New Perfs
** NO GRO
    no "generic-receive-offload",
    no "rx-gro-hw",
*** legacy IPsec - no iptfs
------------------------------------------------------------
Client connecting to 10.0.2.4, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.1 port 52900 connected with 10.0.2.4 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0196 sec  6.17 GBytes  5.29 Gbits/sec
root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1#

root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4
------------------------------------------------------------
Client connecting to 10.0.2.4, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.1 port 49186 connected with 10.0.2.4 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0113 sec  6.32 GBytes  5.42 Gbits/sec
root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1#

*** iptfs

root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4
------------------------------------------------------------
Client connecting to 10.0.2.4, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.1 port 51934 connected with 10.0.2.4 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0103 sec  5.31 GBytes  4.55 Gbits/sec
root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4
------------------------------------------------------------
Client connecting to 10.0.2.4, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.1 port 45320 connected with 10.0.2.4 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0115 sec  5.32 GBytes  4.57 Gbits/sec
root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1#

** GRO
*** legacy IPsec - no iptfs

root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4
------------------------------------------------------------
Client connecting to 10.0.2.4, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.1 port 33940 connected with 10.0.2.4 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0075 sec  9.59 GBytes  8.23 Gbits/sec
root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4
------------------------------------------------------------
Client connecting to 10.0.2.4, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.1 port 45524 connected with 10.0.2.4 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0136 sec  9.04 GBytes  7.76 Gbits/sec
root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4
------------------------------------------------------------
Client connecting to 10.0.2.4, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.1 port 36432 connected with 10.0.2.4 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0125 sec  8.28 GBytes  7.10 Gbits/sec
root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4
------------------------------------------------------------
Client connecting to 10.0.2.4, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.1 port 40362 connected with 10.0.2.4 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0113 sec  8.79 GBytes  7.54 Gbits/sec
root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1#

*** iptfs

root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4
------------------------------------------------------------
Client connecting to 10.0.2.4, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.1 port 58158 connected with 10.0.2.4 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0092 sec  3.52 GBytes  3.03 Gbits/sec
root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4
------------------------------------------------------------
Client connecting to 10.0.2.4, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.1 port 33260 connected with 10.0.2.4 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0095 sec  6.31 GBytes  5.42 Gbits/sec
root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4
------------------------------------------------------------
Client connecting to 10.0.2.4, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.1 port 57584 connected with 10.0.2.4 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0105 sec  5.34 GBytes  4.58 Gbits/sec
root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1# iperf -c 10.0.2.4
------------------------------------------------------------
Client connecting to 10.0.2.4, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.1 port 44142 connected with 10.0.2.4 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0099 sec  4.43 GBytes  3.80 Gbits/sec
root@h1:/tmp/unet-test/tests.iperf.test_iperf_phy/h1#
* Debugging SLAB

  5s IPv6

# name            <active_objs> <objsize> <pagesperslab>       <batchcount> : # slabdata    <num_slabs>
#                        <num_objs> <objperslab> : tunables <limit> # <sharedfactor> <active_slabs>   <sharedavail>

kmalloc-8k           282    294  16384    2    8 : tunables    0    0    0 : slabdata    147    147      0
kmalloc-4k         40861  40980   8192    4    8 : tunables    0    0    0 : slabdata  10245  10245      0
kmalloc-2k           377    528   4096    8    8 : tunables    0    0    0 : slabdata     66     66      0
kmalloc-1k           714    912   2048   16    8 : tunables    0    0    0 : slabdata     57     57      0
kmalloc-512        76535  76688   1024   16    4 : tunables    0    0    0 : slabdata   4793   4793      0
kmalloc-256          880    880    512   16    2 : tunables    0    0    0 : slabdata     55     55      0
kmalloc-192          492    544    256   16    1 : tunables    0    0    0 : slabdata     34     34      0
kmalloc-128          521    592    256   16    1 : tunables    0    0    0 : slabdata     37     37      0
kmalloc-96          6241   6656    128   32    1 : tunables    0    0    0 : slabdata    208    208      0
kmalloc-64          2984   3968    128   32    1 : tunables    0    0    0 : slabdata    124    124      0
kmalloc-32          1873   3328     64   64    1 : tunables    0    0    0 : slabdata     52     52      0
kmalloc-16          1285   2048     32  128    1 : tunables    0    0    0 : slabdata     16     16      0
kmalloc-8           2026   4386     40  102    1 : tunables    0    0    0 : slabdata     43     43      0
kmem_cache_node      288    288    128   32    1 : tunables    0    0    0 : slabdata      9      9      0
kmem_cache           240    240    320   12    1 : tunables    0    0    0 : slabdata     20     20      0
bash-5.2# grep skb /proc/slabinfo
skbuff_ext_cache  118994 166304    128   32    1 : tunables    0    0    0 : slabdata   5197   5197      0
skbuff_small_head    315    336    768   21    4 : tunables    0    0    0 : slabdata     16     16      0
skbuff_fclone_cache    108    108    640   12    2 : tunables    0    0    0 : slabdata      9      9      0
skbuff_head_cache 187671 284244    320   12    1 : tunables    0    0    0 : slabdata  23687  23687      0

15s IPv4

kmalloc-8k           263    274  16384    2    8 : tunables    0    0    0 : slabdata    137    137      0
kmalloc-4k           426    508   8192    4    8 : tunables    0    0    0 : slabdata    127    127      0
kmalloc-2k          8544   8592   4096    8    8 : tunables    0    0    0 : slabdata   1074   1074      0
kmalloc-1k           666    880   2048   16    8 : tunables    0    0    0 : slabdata     55     55      0
kmalloc-512       275412 275504   1024   16    4 : tunables    0    0    0 : slabdata  17219  17219      0
kmalloc-256          782    832    512   16    2 : tunables    0    0    0 : slabdata     52     52      0
kmalloc-192          463    528    256   16    1 : tunables    0    0    0 : slabdata     33     33      0
kmalloc-128          514    576    256   16    1 : tunables    0    0    0 : slabdata     36     36      0
kmalloc-96         13940  14080    128   32    1 : tunables    0    0    0 : slabdata    440    440      0
kmalloc-64          3077   4000    128   32    1 : tunables    0    0    0 : slabdata    125    125      0
kmalloc-32          1672   2944     64   64    1 : tunables    0    0    0 : slabdata     46     46      0
kmalloc-16          1198   2176     32  128    1 : tunables    0    0    0 : slabdata     17     17      0
kmalloc-8           1892   4182     40  102    1 : tunables    0    0    0 : slabdata     41     41      0
kmem_cache_node      256    256    128   32    1 : tunables    0    0    0 : slabdata      8      8      0
kmem_cache           240    240    320   12    1 : tunables    0    0    0 : slabdata     20     20      0
bash-5.2# grep skb /proc/slabinfo
skbuff_ext_cache  273929 275136    128   32    1 : tunables    0    0    0 : slabdata   8598   8598      0
skbuff_small_head  50568  50589    768   21    4 : tunables    0    0    0 : slabdata   2409   2409      0
skbuff_fclone_cache     96     96    640   12    2 : tunables    0    0    0 : slabdata      8      8      0
skbuff_head_cache 175134 175716    320   12    1 : tunables    0    0    0 : slabdata  14643  14643      0
b

** Why policy route failing
   * Unknown packet hist ip_route_input_noref
   * ip_route_input_rcu(struct sk_buff *skb, __be32 daddr, __be32 saddr,
   * ip_route_input_slow(struct sk_buff *skb, __be32 daddr, __be32 saddr,


#0  ip_route_input_slow (skb=0xffff888104718b00, daddr=<optimized out>, saddr=<optimized out>, tos=tos@entry=0 '\000',
    dev=0xffff888102a74000, res=0xffffc90000003b90) at /home/chopps/w/iptfs-dev/linux/net/ipv4/route.c:2303
#1  0xffffffff81e8cfe3 in ip_route_input_rcu (skb=skb@entry=0xffff888104718b00, daddr=daddr@entry=84082698, saddr=saddr@entry=16777226,
    tos=tos@entry=0 '\000', dev=dev@entry=0xffff888102a74000, res=res@entry=0xffffc90000003b90)
    at /home/chopps/w/iptfs-dev/linux/net/ipv4/route.c:2481
#2  0xffffffff81e8d086 in ip_route_input_noref (skb=skb@entry=0xffff888104718b00, daddr=84082698, saddr=16777226, tos=<optimized out>,
    dev=dev@entry=0xffff888102a74000) at /home/chopps/w/iptfs-dev/linux/net/ipv4/route.c:2492
#3  0xffffffff81e8fa1a in ip_rcv_finish_core (net=net@entry=0xffffffff833e60c0 <init_net>, skb=skb@entry=0xffff888104718b00,
    dev=0xffff888102a74000, hint=hint@entry=0x0 <fixed_percpu_data>, sk=0x0 <fixed_percpu_data>)
    at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_input.c:365
#
*** working from strongswan

    #0  __xfrm_route_forward (skb=skb@entry=0xffff888104c3c600, family=family@entry=2) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_policy.c:3794
    #1  0xffffffff81e9241a in xfrm_route_forward (family=2, skb=0xffff888104c3c600) at /home/chopps/w/iptfs-dev/linux/include/net/xfrm.h:1273
    #2  xfrm4_route_forward (skb=0xffff888104c3c600) at /home/chopps/w/iptfs-dev/linux/include/net/xfrm.h:1278
    #3  ip_forward (skb=0xffff888104c3c600) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_forward.c:124
    #4  0xffffffff81e9080f in dst_input (skb=0xffff888104e70501) at /home/chopps/w/iptfs-dev/linux/include/net/dst.h:468
    #5  ip_sublist_rcv_finish (head=head@entry=0xffffc90000003c48) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_input.c:580
    #6  0xffffffff81e90946 in ip_list_rcv_finish (sk=0x0 <fixed_percpu_data>, head=0xffffc90000003cd0, net=0xffffffff833e60c0 <init_net>)
        at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_input.c:630
    #7  ip_sublist_rcv (head=head@entry=0xffffc90000003cd0, dev=dev@entry=0xffff888102f24000, net=net@entry=0xffffffff833e60c0 <init_net>)
        at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_input.c:638
    #8  0xffffffff81e90cbd in ip_list_rcv (head=0xffffc90000003d50, pt=<optimized out>, orig_dev=<optimized out>)
        at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_input.c:673
    #9  0xffffffff81dd8dc8 in __netif_receive_skb_list_ptype (orig_dev=0xffff888102f24000, pt_prev=0xffffffff82ec4e60 <ip_packet_type>, head=0xffffc90000003d50)
        at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5534
    #10 __netif_receive_skb_list_core (head=head@entry=0xffff888102940110, pfmemalloc=pfmemalloc@entry=false) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5582
    #11 0xffffffff81dd93cf in __netif_receive_skb_list (head=0xffff888102940110) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5634
    #12 netif_receive_skb_list_internal (head=head@entry=0xffff888102940110) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5725
    #13 0xffffffff81dd9b7e in gro_normal_list (napi=0xffff888102940008) at /home/chopps/w/iptfs-dev/linux/include/net/gro.h:433


sh-5.2# ip -d route list table 220
unicast 10.0.0.0/24 dev eth2 proto static scope link src 10.0.3.4

unicast 10.0.3.0/24 via 10.0.1.3 dev eth2 proto static scope global src 10.0.0.2

sh-5.2# ip -d  route list table 220
unicast 10.0.0.0/24 via 10.0.2.3 dev eth2 proto static scope global src 10.0.3.4


BAD:
------ Host: r1 ------
unicast 10.0.3.0/24 via 10.0.1.3 dev eth2 proto static scope global
------- End: r1 ------
------ Host: r2 ------
unicast 10.0.0.0/24 via 10.0.2.3 dev eth2 proto static scope global
------- End: r2 ------

munet> sh ip -d route list table main
------ Host: h1 ------
unicast 10.0.0.0/24 dev eth1 proto kernel scope link src 10.0.0.1
unicast 10.0.1.0/24 via 10.0.0.2 dev eth1 proto boot scope global
unicast 10.0.2.0/24 via 10.0.0.2 dev eth1 proto boot scope global
unicast 10.0.3.0/24 via 10.0.0.2 dev eth1 proto boot scope global
unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.1
------- End: h1 ------
------ Host: h2 ------
unicast 10.0.0.0/24 via 10.0.3.4 dev eth1 proto boot scope global
unicast 10.0.1.0/24 via 10.0.3.4 dev eth1 proto boot scope global
unicast 10.0.2.0/24 via 10.0.3.4 dev eth1 proto boot scope global
unicast 10.0.3.0/24 dev eth1 proto kernel scope link src 10.0.3.5
unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.5
------- End: h2 ------
------ Host: r1 ------
unicast 10.0.0.0/24 dev eth1 proto kernel scope link src 10.0.0.2
unicast 10.0.1.0/24 dev eth2 proto kernel scope link src 10.0.1.2
unicast 10.0.2.4 via 10.0.1.3 dev eth2 proto boot scope global
unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.2
------- End: r1 ------
------ Host: r2 ------
unicast 10.0.1.2 via 10.0.2.3 dev eth2 proto boot scope global
unicast 10.0.2.0/24 dev eth2 proto kernel scope link src 10.0.2.4
unicast 10.0.3.0/24 dev eth1 proto kernel scope link src 10.0.3.4
unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.4
------- End: r2 ------
------ Host: rm ------
unicast 10.0.1.0/24 dev eth1 proto kernel scope link src 10.0.1.3
unicast 10.0.2.0/24 dev eth2 proto kernel scope link src 10.0.2.3
unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.3
------- End: rm ------


GOOD:
------ Host: r1 ------
unicast 10.0.3.0/24 via 10.0.1.3 dev eth2 proto static scope global src 10.0.0.2
unicast 12.12.12.0/24 via 10.0.1.3 dev eth2 proto static scope global src 10.0.0.2
------- End: r1 ------
------ Host: r2 ------
unicast 10.0.0.0/24 via 10.0.2.3 dev eth2 proto static scope global src 10.0.3.4
unicast 11.11.11.0/24 via 10.0.2.3 dev eth2 proto static scope global src 10.0.3.4
------- End: r2 ------

munet> sh ip -d route list table main
------ Host: h1 ------
unicast 10.0.0.0/24 dev eth1 proto kernel scope link src 10.0.0.1
unicast 10.0.1.0/24 via 10.0.0.2 dev eth1 proto boot scope global
unicast 10.0.2.0/24 via 10.0.0.2 dev eth1 proto boot scope global
unicast 10.0.3.0/24 via 10.0.0.2 dev eth1 proto boot scope global
unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.1
------- End: h1 ------
------ Host: h2 ------
unicast 10.0.0.0/24 via 10.0.3.4 dev eth1 proto boot scope global
unicast 10.0.1.0/24 via 10.0.3.4 dev eth1 proto boot scope global
unicast 10.0.2.0/24 via 10.0.3.4 dev eth1 proto boot scope global
unicast 10.0.3.0/24 dev eth1 proto kernel scope link src 10.0.3.5
unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.5
------- End: h2 ------
------ Host: r1 ------
unicast 10.0.0.0/24 dev eth1 proto kernel scope link src 10.0.0.2
unicast 10.0.1.0/24 dev eth2 proto kernel scope link src 10.0.1.2
unicast 10.0.2.0/24 via 10.0.1.3 dev eth2 proto boot scope global
unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.2
------- End: r1 ------
------ Host: r2 ------
unicast 10.0.1.0/24 via 10.0.2.3 dev eth2 proto boot scope global
unicast 10.0.2.0/24 dev eth2 proto kernel scope link src 10.0.2.4
unicast 10.0.3.0/24 dev eth1 proto kernel scope link src 10.0.3.4
unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.4
------- End: r2 ------
------ Host: rm ------
unicast 10.0.1.0/24 dev eth1 proto kernel scope link src 10.0.1.3
unicast 10.0.2.0/24 dev eth2 proto kernel scope link src 10.0.2.3
unicast 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.3
------- End: rm ------

*** BAD FULLuTABLE
munet> sh ip -d route list table all
------ Host: h1 ------
unicast 10.0.0.0/24 dev eth1 table main proto kernel scope link src 10.0.0.1
unicast 10.0.1.0/24 via 10.0.0.2 dev eth1 table main proto boot scope global
unicast 10.0.2.0/24 via 10.0.0.2 dev eth1 table main proto boot scope global
unicast 10.0.3.0/24 via 10.0.0.2 dev eth1 table main proto boot scope global
unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.1
local 10.0.0.1 dev eth1 table local proto kernel scope host src 10.0.0.1
broadcast 10.0.0.255 dev eth1 table local proto kernel scope link src 10.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 192.168.0.1 dev eth0 table local proto kernel scope host src 192.168.0.1
broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.1
------- End: h1 ------
------ Host: h2 ------
unicast 10.0.0.0/24 via 10.0.3.4 dev eth1 table main proto boot scope global
unicast 10.0.1.0/24 via 10.0.3.4 dev eth1 table main proto boot scope global
unicast 10.0.2.0/24 via 10.0.3.4 dev eth1 table main proto boot scope global
unicast 10.0.3.0/24 dev eth1 table main proto kernel scope link src 10.0.3.5
unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.5
local 10.0.3.5 dev eth1 table local proto kernel scope host src 10.0.3.5
broadcast 10.0.3.255 dev eth1 table local proto kernel scope link src 10.0.3.5
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 192.168.0.5 dev eth0 table local proto kernel scope host src 192.168.0.5
broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.5
------- End: h2 ------
------ Host: r1 ------
unicast 10.0.3.0/24 via 10.0.1.3 dev eth2 table 220 proto static scope global src 10.0.0.2

unicast 10.0.0.0/24 dev eth1 table main proto kernel scope link src 10.0.0.2
unicast 10.0.1.0/24 dev eth2 table main proto kernel scope link src 10.0.1.2
unicast 10.0.2.4 via 10.0.1.3 dev eth2 table main proto boot scope global
unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.2

local 10.0.0.2 dev eth1 table local proto kernel scope host src 10.0.0.2
broadcast 10.0.0.255 dev eth1 table local proto kernel scope link src 10.0.0.2
local 10.0.1.2 dev eth2 table local proto kernel scope host src 10.0.1.2
broadcast 10.0.1.255 dev eth2 table local proto kernel scope link src 10.0.1.2
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 192.168.0.2 dev eth0 table local proto kernel scope host src 192.168.0.2
broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.2
------- End: r1 ------
------ Host: r2 ------
unicast 10.0.0.0/24 via 10.0.2.3 dev eth2 table 220 proto static scope global src 10.0.3.4
unicast 10.0.1.2 via 10.0.2.3 dev eth2 table main proto boot scope global
unicast 10.0.2.0/24 dev eth2 table main proto kernel scope link src 10.0.2.4
unicast 10.0.3.0/24 dev eth1 table main proto kernel scope link src 10.0.3.4
unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.4
local 10.0.2.4 dev eth2 table local proto kernel scope host src 10.0.2.4
broadcast 10.0.2.255 dev eth2 table local proto kernel scope link src 10.0.2.4
local 10.0.3.4 dev eth1 table local proto kernel scope host src 10.0.3.4
broadcast 10.0.3.255 dev eth1 table local proto kernel scope link src 10.0.3.4
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 192.168.0.4 dev eth0 table local proto kernel scope host src 192.168.0.4
broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.4
------- End: r2 ------
------ Host: rm ------
unicast 10.0.1.0/24 dev eth1 table main proto kernel scope link src 10.0.1.3
unicast 10.0.2.0/24 dev eth2 table main proto kernel scope link src 10.0.2.3
unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.3
local 10.0.1.3 dev eth1 table local proto kernel scope host src 10.0.1.3
broadcast 10.0.1.255 dev eth1 table local proto kernel scope link src 10.0.1.3
local 10.0.2.3 dev eth2 table local proto kernel scope host src 10.0.2.3
broadcast 10.0.2.255 dev eth2 table local proto kernel scope link src 10.0.2.3
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 192.168.0.3 dev eth0 table local proto kernel scope host src 192.168.0.3
broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.3
------- End: rm ------
munet>

GOOD

munet> sh ip -d route list table all
------ Host: h1 ------
unicast 10.0.0.0/24 dev eth1 table main proto kernel scope link src 10.0.0.1
unicast 10.0.1.0/24 via 10.0.0.2 dev eth1 table main proto boot scope global
unicast 10.0.2.0/24 via 10.0.0.2 dev eth1 table main proto boot scope global
unicast 10.0.3.0/24 via 10.0.0.2 dev eth1 table main proto boot scope global
unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.1
local 10.0.0.1 dev eth1 table local proto kernel scope host src 10.0.0.1
broadcast 10.0.0.255 dev eth1 table local proto kernel scope link src 10.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 192.168.0.1 dev eth0 table local proto kernel scope host src 192.168.0.1
broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.1
------- End: h1 ------
------ Host: h2 ------
unicast 10.0.0.0/24 via 10.0.3.4 dev eth1 table main proto boot scope global
unicast 10.0.1.0/24 via 10.0.3.4 dev eth1 table main proto boot scope global
unicast 10.0.2.0/24 via 10.0.3.4 dev eth1 table main proto boot scope global
unicast 10.0.3.0/24 dev eth1 table main proto kernel scope link src 10.0.3.5
unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.5
local 10.0.3.5 dev eth1 table local proto kernel scope host src 10.0.3.5
broadcast 10.0.3.255 dev eth1 table local proto kernel scope link src 10.0.3.5
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 192.168.0.5 dev eth0 table local proto kernel scope host src 192.168.0.5
broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.5
------- End: h2 ------
------ Host: r1 ------
unicast 10.0.3.0/24 via 10.0.1.3 dev eth2 table 220 proto static scope global src 10.0.0.2
unicast 12.12.12.0/24 via 10.0.1.3 dev eth2 table 220 proto static scope global src 10.0.0.2

unicast 10.0.0.0/24 dev eth1 table main proto kernel scope link src 10.0.0.2
unicast 10.0.1.0/24 dev eth2 table main proto kernel scope link src 10.0.1.2
unicast 10.0.2.0/24 via 10.0.1.3 dev eth2 table main proto boot scope global
unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.2

local 10.0.0.2 dev eth1 table local proto kernel scope host src 10.0.0.2
broadcast 10.0.0.255 dev eth1 table local proto kernel scope link src 10.0.0.2
local 10.0.1.2 dev eth2 table local proto kernel scope host src 10.0.1.2
broadcast 10.0.1.255 dev eth2 table local proto kernel scope link src 10.0.1.2
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 192.168.0.2 dev eth0 table local proto kernel scope host src 192.168.0.2
broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.2
------- End: r1 ------
------ Host: r2 ------
unicast 10.0.0.0/24 via 10.0.2.3 dev eth2 table 220 proto static scope global src 10.0.3.4
unicast 11.11.11.0/24 via 10.0.2.3 dev eth2 table 220 proto static scope global src 10.0.3.4
unicast 10.0.1.0/24 via 10.0.2.3 dev eth2 table main proto boot scope global
unicast 10.0.2.0/24 dev eth2 table main proto kernel scope link src 10.0.2.4
unicast 10.0.3.0/24 dev eth1 table main proto kernel scope link src 10.0.3.4
unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.4
local 10.0.2.4 dev eth2 table local proto kernel scope host src 10.0.2.4
broadcast 10.0.2.255 dev eth2 table local proto kernel scope link src 10.0.2.4
local 10.0.3.4 dev eth1 table local proto kernel scope host src 10.0.3.4
broadcast 10.0.3.255 dev eth1 table local proto kernel scope link src 10.0.3.4
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 192.168.0.4 dev eth0 table local proto kernel scope host src 192.168.0.4
broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.4
------- End: r2 ------
------ Host: rm ------
unicast 10.0.1.0/24 dev eth1 table main proto kernel scope link src 10.0.1.3
unicast 10.0.2.0/24 dev eth2 table main proto kernel scope link src 10.0.2.3
unicast 192.168.0.0/24 dev eth0 table main proto kernel scope link src 192.168.0.3
local 10.0.1.3 dev eth1 table local proto kernel scope host src 10.0.1.3
broadcast 10.0.1.255 dev eth1 table local proto kernel scope link src 10.0.1.3
local 10.0.2.3 dev eth2 table local proto kernel scope host src 10.0.2.3
broadcast 10.0.2.255 dev eth2 table local proto kernel scope link src 10.0.2.3
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 192.168.0.3 dev eth0 table local proto kernel scope host src 192.168.0.3
broadcast 192.168.0.255 dev eth0 table local proto kernel scope link src 192.168.0.3
------- End: rm ------

* features/sharing gso/gro_cells
(py39) [0s][04:14:18 labnh:~/w/iptfs-dev]$ python scripts/feat2str.py 0x201901fd54bb3
['SG_BIT', 'IP_CSUM_BIT', 'IPV6_CSUM_BIT', 'HIGHDMA_BIT', 'HW_VLAN_CTAG_TX_BIT',
'HW_VLAN_CTAG_RX_BIT', 'HW_VLAN_CTAG_FILTER_BIT', 'GSO_BIT', 'GRO_BIT',

'TSO_BIT', 'TSO_ECN_BIT', 'TSO6_BIT', 'GSO_GRE_BIT', 'GSO_GRE_CSUM_BIT',
'GSO_IPXIP4_BIT', 'GSO_IPXIP6_BIT', 'GSO_UDP_TUNNEL_BIT',
'GSO_UDP_TUNNEL_CSUM_BIT', 'GSO_PARTIAL_BIT', 'SCTP_CRC_BIT', 'RXHASH_BIT',
'RXCSUM_BIT', 'HW_TC_BIT']

# This is missing {GRE,UDP}_TUNNEL_CSUM_BIT's

(py39) [0s][04:56:02 labnh:~/w/iptfs-dev]$ python scripts/feat2str.py 0x2019017554bb3
['SG_BIT', 'IP_CSUM_BIT', 'IPV6_CSUM_BIT', 'HIGHDMA_BIT', 'HW_VLAN_CTAG_TX_BIT',
'HW_VLAN_CTAG_RX_BIT', 'HW_VLAN_CTAG_FILTER_BIT', 'GSO_BIT', 'GRO_BIT',

'TSO_BIT', 'TSO_ECN_BIT', 'TSO6_BIT', 'GSO_GRE_BIT',
'GSO_IPXIP4_BIT', 'GSO_IPXIP6_BIT', 'GSO_UDP_TUNNEL_BIT',
                           'GSO_PARTIAL_BIT', 'SCTP_CRC_BIT', 'RXHASH_BIT',
 'RXCSUM_BIT', 'HW_TC_BIT']
* packets at egress:
** GRO/GSO enabled
*** PHY
head_frag == 1, nr_frags == 0, end == 1728, headroom == 242 (data)

**** R2
##### CPU 1 buffer started ####
     ksoftirqd/1-20      [001] ..s..     9.540379: iptfs_egress_recv: EGRESS: skb=000000006d1fa058 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0
		tailroom=40 tail=1688 end=1728 head=ffff88810ea5a800 hdpgaddr=ffff88810ea5a000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0

     ksoftirqd/1-20      [001] ..s..     9.540402: iptfs_egress_recv: EGRESS: skb=00000000156d5d79 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=7558
		tailroom=40 tail=1688 end=1728 head=ffff88810ea5b800 hdpgaddr=ffff88810ea5b000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/1-20      [001] ..s..     9.540419: iptfs_egress_recv: EGRESS: skb=00000000c70110b1 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=6116
		tailroom=40 tail=1688 end=1728 head=ffff88810ea5c800 hdpgaddr=ffff88810ea5c000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/1-20      [001] ..s..     9.540436: iptfs_egress_recv: EGRESS: skb=000000001ed1d59d len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=4674
		tailroom=40 tail=1688 end=1728 head=ffff88810ea5d800 hdpgaddr=ffff88810ea5d000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/1-20      [001] ..s..     9.540453: iptfs_egress_recv: EGRESS: skb=00000000eb724e7f len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=3232
		tailroom=40 tail=1688 end=1728 head=ffff88810ea5e800 hdpgaddr=ffff88810ea5e000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/1-20      [001] ..s..     9.540470: iptfs_egress_recv: EGRESS: skb=0000000014591435 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=1790
		tailroom=40 tail=1688 end=1728 head=ffff88810ea5f800 hdpgaddr=ffff88810ea5f000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/1-20      [001] ..s..     9.540487: iptfs_egress_recv: EGRESS: skb=000000000ad8eb58 len=352 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=348
		tailroom=1134 tail=594 end=1728 head=ffff88810ea60800 hdpgaddr=ffff88810ea60000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/1-20      [001] ..s..     9.540505: iptfs_egress_recv: EGRESS: skb=000000002453d179 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0
		tailroom=40 tail=1688 end=1728 head=ffff88810ea61800 hdpgaddr=ffff88810ea61000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
*** NON
**** R1
     ksoftirqd/0-14      [000] ..s1.     8.144958: iptfs_egress_recv: EGRESS: skb=000000001d284ad6 len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=0
		tailroom=0 tail=114 end=704 head=ffff88801b667800 hdpgaddr=ffff88801b667000 pg0->addr=ffff888011520000 pg0->data=ffff88801152603e pg0->off=24638
     ksoftirqd/0-14      [000] ..s1.     8.144981: iptfs_egress_recv: EGRESS: skb=000000007791ad14 len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=7558
		tailroom=0 tail=114 end=704 head=ffff88801b667400 hdpgaddr=ffff88801b667000 pg0->addr=ffff888011520000 pg0->data=ffff88801152663e pg0->off=26174
     ksoftirqd/0-14      [000] ..s1.     8.145119: iptfs_egress_recv: EGRESS: skb=000000004969634f len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=6116
		tailroom=0 tail=114 end=704 head=ffff88801b66e400 hdpgaddr=ffff88801b66e000 pg0->addr=ffff888011520000 pg0->data=ffff888011526c3e pg0->off=27710
     ksoftirqd/0-14      [000] ..s1.     8.145140: iptfs_egress_recv: EGRESS: skb=00000000315dd6bd len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=4674
		tailroom=0 tail=114 end=704 head=ffff88801b66e000 hdpgaddr=ffff88801b66e000 pg0->addr=ffff888011520000 pg0->data=ffff88801152723e pg0->off=29246
     ksoftirqd/0-14      [000] ..s..     8.145163: iptfs_egress_recv: EGRESS: skb=00000000695e660c len=1446 data_len=0 headroom=62 head_frag=1 frag_list=0 nr_frags=0 blkoff=3232
		tailroom=220 tail=1508 end=1728 head=ffff888011527800 hdpgaddr=ffff888011520000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/0-14      [000] ..s1.     8.145184: iptfs_egress_recv: EGRESS: skb=00000000702a4cee len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=1790
		tailroom=0 tail=114 end=704 head=ffff88801b673c00 hdpgaddr=ffff88801b673000 pg0->addr=ffff8880119e8000 pg0->data=ffff8880119e803e pg0->off=62
     ksoftirqd/0-14      [000] ..s..     8.145206: iptfs_egress_recv: EGRESS: skb=00000000f3979ed2 len=352 data_len=0 headroom=62 head_frag=1 frag_list=0 nr_frags=0 blkoff=348
		tailroom=802 tail=414 end=1216 head=ffff8880119e8600 hdpgaddr=ffff8880119e8000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/0-14      [000] ..s1.     8.145225: iptfs_egress_recv: EGRESS: skb=000000005abe0cf1 len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=0
		tailroom=0 tail=114 end=704 head=ffff88801b673800 hdpgaddr=ffff88801b673000 pg0->addr=ffff8880119e8000 pg0->data=ffff8880119e8c3e pg0->off=3134
** GRO/GSO disabled
*** R1
**** NON: small packets with a small amounts of aggregation it seems

head_frag == 1, nr_frags == 0, end == 1728, headroom == 242 (data)

ksoftirqd/0-14      [000] ..s..     8.006165: iptfs_egress_recv: EGRESS: skb=00000000345e5161 len=368 data_len=0 headroom=62 head_frag=1 frag_list=0 nr_frags=0 blkoff=0
		tailroom=786 tail=430 end=1216 head=ffff8880212db000 hdpgaddr=ffff8880212d8000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/0-14      [000] ..s..     8.006240: iptfs_egress_recv: EGRESS: skb=00000000fec9351c len=56 data_len=0 headroom=114 head_frag=1 frag_list=0 nr_frags=0 blkoff=0
		tailroom=534 tail=170 end=704 head=ffff888009cd4000 hdpgaddr=ffff888009cd4000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/0-14      [000] ..s..     8.006254: iptfs_egress_recv: EGRESS: skb=000000009a3b8aab len=56 data_len=0 headroom=114 head_frag=1 frag_list=0 nr_frags=0 blkoff=0
		tailroom=534 tail=170 end=704 head=ffff888009926c00 hdpgaddr=ffff888009926000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/0-14      [000] ..s..     8.006268: iptfs_egress_recv: EGRESS: skb=0000000083f12223 len=108 data_len=0 headroom=62 head_frag=1 frag_list=0 nr_frags=0 blkoff=0
		tailroom=1046 tail=170 end=1216 head=ffff8880212dc200 hdpgaddr=ffff8880212d8000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0

NON: [000] ..s1.    15.334429: iptfs_egress_recv: EGRESS: skb=00000000029a7314 len=1412 data_len=1412 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=0 tailroom=0 tail=114 end=704 head=ffff888019c8c400 hdpgaddr=ffff888019c8c000 pg0->addr=ffff888020280000 pg0->data=ffff88802028243e pg0->off=9278

**** PHY
PHY: [001] ..s1.     7.412970: iptfs_egress_recv: EGRESS: skb=0000000043420e6b len=212 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0 tailroom=1274 tail=454 end=1728 head=ffff88810eb0e000 hdpgaddr=ffff88810eb0e000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0

     ksoftirqd/0-14      [000] ..s..     8.411730: iptfs_egress_recv: EGRESS: skb=00000000de2755bc len=56 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0
		tailroom=1430 tail=298 end=1728 head=ffff88810eb6e000 hdpgaddr=ffff88810eb6e000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0

     ksoftirqd/0-14      [000] ..s..     8.411745: iptfs_egress_recv: EGRESS: skb=00000000dcac57c7 len=56 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0
		tailroom=1430 tail=298 end=1728 head=ffff88810eb6f000 hdpgaddr=ffff88810eb6f000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/0-14      [000] ..s..     8.411759: iptfs_egress_recv: EGRESS: skb=00000000e5e49dd8 len=56 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0
		tailroom=1430 tail=298 end=1728 head=ffff88810eb70000 hdpgaddr=ffff88810eb70000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/0-14      [000] ..s..     8.411774: iptfs_egress_recv: EGRESS: skb=000000002f5710b4 len=56 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0
		tailroom=1430 tail=298 end=1728 head=ffff88810eb71000 hdpgaddr=ffff88810eb71000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/0-14      [000] ..s..     8.411788: iptfs_egress_recv: EGRESS: skb=00000000ca57c05a len=56 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0
		tailroom=1430 tail=298 end=1728 head=ffff88810eb72000 hdpgaddr=ffff88810eb72000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/0-14      [000] ..s..     8.411802: iptfs_egress_recv: EGRESS: skb=00000000d813b429 len=108 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0
		tailroom=1378 tail=350 end=1728 head=ffff88810eb73000 hdpgaddr=ffff88810eb73000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0

*** R2
**** NON:

     ksoftirqd/0-14      [000] ..s1.     8.180352: iptfs_egress_recv: EGRESS: skb=00000000acf45ccb len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=0
		tailroom=0 tail=114 end=704 head=ffff88801a90b400 hdpgaddr=ffff88801a90b000 pg0->addr=ffff888012148000 pg0->data=ffff88801214da3e pg0->off=23102

     ksoftirqd/0-14      [000] ..s1.     8.180379: iptfs_egress_recv: EGRESS: skb=0000000097b1ffcb len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=7558
		tailroom=0 tail=114 end=704 head=ffff88801a90b000 hdpgaddr=ffff88801a90b000 pg0->addr=ffff888012148000 pg0->data=ffff88801214e03e pg0->off=24638

     ksoftirqd/0-14      [000] ..s1.     8.180481: iptfs_egress_recv: EGRESS: skb=0000000090088909 len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=6116
		tailroom=0 tail=114 end=704 head=ffff88801a90b800 hdpgaddr=ffff88801a90b000 pg0->addr=ffff888012148000 pg0->data=ffff88801214e63e pg0->off=26174

     ksoftirqd/0-14      [000] ..s1.     8.180503: iptfs_egress_recv: EGRESS: skb=00000000cc18d723 len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=4674
		tailroom=0 tail=114 end=704 head=ffff88801a90b400 hdpgaddr=ffff88801a90b000 pg0->addr=ffff888012148000 pg0->data=ffff88801214ec3e pg0->off=27710

     ksoftirqd/0-14      [000] ..s1.     8.180525: iptfs_egress_recv: EGRESS: skb=000000008a12301d len=1446 data_len=1446 headroom=114 head_frag=1 frag_list=0 nr_frags=1 blkoff=3232
		tailroom=0 tail=114 end=704 head=ffff88801a90b000 hdpgaddr=ffff88801a90b000 pg0->addr=ffff888012148000 pg0->data=ffff88801214f23e pg0->off=29246

     ksoftirqd/0-14      [000] ..s..     8.180546: iptfs_egress_recv: EGRESS: skb=00000000e59cfe58 len=1446 data_len=0 headroom=62 head_frag=1 frag_list=0 nr_frags=0 blkoff=1790
		tailroom=220 tail=1508 end=1728 head=ffff88801214f800 hdpgaddr=ffff888012148000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0

     ksoftirqd/0-14      [000] ..s..     8.180574: iptfs_egress_recv: EGRESS: skb=0000000079b524b0 len=352 data_len=0 headroom=62 head_frag=1 frag_list=0 nr_frags=0 blkoff=348
		tailroom=802 tail=414 end=1216 head=ffff888011998000 hdpgaddr=ffff888011998000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0

**** PHY
     ksoftirqd/1-20      [001] ..s..     7.647452: iptfs_egress_recv: EGRESS: skb=0000000006bf1593 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=0
		tailroom=40 tail=1688 end=1728 head=ffff88810ea02000 hdpgaddr=ffff88810ea02000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/1-20      [001] ..s..     7.647479: iptfs_egress_recv: EGRESS: skb=00000000b9ac19cc len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=7558
		tailroom=40 tail=1688 end=1728 head=ffff88810ea03000 hdpgaddr=ffff88810ea03000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/1-20      [001] ..s..     7.647500: iptfs_egress_recv: EGRESS: skb=00000000bac669ce len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=6116
		tailroom=40 tail=1688 end=1728 head=ffff88810ea04000 hdpgaddr=ffff88810ea04000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/1-20      [001] ..s..     7.647578: iptfs_egress_recv: EGRESS: skb=0000000052a43342 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=4674
		tailroom=40 tail=1688 end=1728 head=ffff88810ea05000 hdpgaddr=ffff88810ea05000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/1-20      [001] ..s..     7.647599: iptfs_egress_recv: EGRESS: skb=000000009e3c3cb2 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=3232
		tailroom=40 tail=1688 end=1728 head=ffff88810ea06000 hdpgaddr=ffff88810ea06000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/1-20      [001] ..s..     7.647620: iptfs_egress_recv: EGRESS: skb=00000000afd492e1 len=1446 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=1790
		tailroom=40 tail=1688 end=1728 head=ffff88810ea07000 hdpgaddr=ffff88810ea07000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0
     ksoftirqd/1-20      [001] ..s..     7.647643: iptfs_egress_recv: EGRESS: skb=000000009e9f8574 len=352 data_len=0 headroom=242 head_frag=1 frag_list=0 nr_frags=0 blkoff=348
		tailroom=1134 tail=594 end=1728 head=ffff88810ea08000 hdpgaddr=ffff88810ea08000 pg0->addr=0000000000000000 pg0->data=0000000000000000 pg0->off=0