Need offical documentation and script examples for collecting BGP routes #262

victor-sudakov · 2015-05-15T08:51:18Z

There is neither documentation nor script examples for collecting/saving BGP routes for further processing (like building src/dst ACLs for squid for example). I would like some to be provided in the distribution.

pyke369 · 2015-05-15T12:22:25Z

Hello Victor,

You may want to take a look at https://github.com/pyke369/exabgp-helpers/tree/master/exaprefixdb. This little ExaBGP helper script will actually dump the routes/prefixes received from an arbitrary number of BGP sessions in multiple files at a regular interval, relying on filters like ASPath, BGP community or BGP peers. It actually works fine with multiples copies of the global Internet public table, received from multiple routers with multiple Tier-1 transit links.

I will add some documentation later, but there is an example configuration file (JSON-based) along with the corresponding ExaBGP configuration file. You may then export the generated text files into fast-lookup databases using https://github.com/pyke369/prefixdb for instance (this comes handy for SDN/applicative routing management).

Cheers.

thomas-mangin · 2015-05-16T09:24:06Z

@pyke369 how about integrating this tool within the exabgp repository ? I would be fine giving you commit access to be able to maintain the code.

victor-sudakov · 2015-05-16T10:47:05Z

@pyke369 Are you sure there is no error in line 10 of exaprefixdb-sample.conf? I am almost sure there should be no comma there.

pyke369 · 2015-05-16T12:35:18Z

@victor-sudakov yes, you are absolutely right:

$ cat exaprefixdb-sample.conf | python -mjson.tool
Expecting property name: line 11 column 4 (char 234)

I fixed it in the repository. Please note this already is an advanced configuration (with 4 BGP sessions and a lot of filtering). The simplest one you could start with is more like:

{
   "exports":
   [
      { "database": "all" }
   ]
}

This will dump all received routes from all established BGP sessions in /tmp/all.db every 10 minutes. You could start with this and then introduce more complexity if needed.

pyke369 · 2015-05-16T12:36:39Z

@thomas-mangin sure thing. let me just clean the repo. a little bit and push a small documentation first.

victor-sudakov · 2015-05-16T14:27:33Z

@pyke369 I'm trying to test it with exabgp-3.4.8 with a simple config (see below) but my database comes out empty all the time. Any ideas why?

exaprefixdb.conf:
{
"interval": 180,
"backups": 5,
"basepath": "/var/tmp/prefixdb",
"aliases":
{
"212.73.125.217": "gateway",
"212.73.125.216": "leopard"
},
"exports":
[
{ "database": "all" }
]
}

exabgp.conf:
process mkdb {
encoder json;
receive {
parsed;
update;
}
run /usr/local/etc/exabgp/exaprefixdb.py /usr/local/etc/exabgp/exaprefixdb.conf;
}

pyke369 · 2015-05-16T16:28:17Z

@victor-sudakov could you check the following please?

the exaprefixdb.py is being started by exabgp and running properly
the user running the exaprefixdb.py process has write access to the the /var/tmp/prefixdb folder
you waited at least seconds (180 seconds in your case, we wait that long to make sure the first dump is not empty or partial, especially with Internet full-tables)
your BGP peers are actually sending routes updates

You may try to start ExaBGP in foreground mode ("exabgp_log_all=true exabgp exabgp.conf") and check the messages on the screen (search for routes updates). In addition, try to "kill -USR1 <exaprefixdb.py PID>" to activate additional syslog debug messages.

For instance, this is the trace I get from a peer announcing the v4 default and private blocks routes:

Sat, 16 May 2015 18:45:43 | INFO     | 17566  | reactor       | New peer setup: neighbor 10.140.0.1 local-ip 192.168.37.128 local-as 65501 peer-as 65500 router-id 192.168.37.128 family-allowed in-open
Sat, 16 May 2015 18:45:43 | WARNING  | 17566  | configuration | Loaded new configuration successfully
Sat, 16 May 2015 18:45:43 | INFO     | 17566  | processes     | Forked process mkdb
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   >> OPEN version=4 asn=65501 hold_time=180 router_id=192.168.37.128 capabilities=[Multiprotocol(ipv4 unicast), ASN4(65501)]
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   << OPEN
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   << OPEN version=4 asn=65500 hold_time=6 router_id=10.128.0.103 capabilities=[Multiprotocol(ipv4 unicast), Route Refresh, Graceful Restart Flags 0x0 Time 120 , ASN4(65500), Route Refresh]
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   >> KEEPALIVE (OPENCONFIRM)
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   << KEEPALIVE
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | network       | Connected to peer neighbor 10.140.0.1 local-ip 192.168.37.128 local-as 65501 peer-as 65500 router-id 192.168.37.128 family-allowed in-open (out)
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   << KEEPALIVE
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   >> EOR ipv4 unicast
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | message       | peer 10.140.0.1 ASN 65500   >> EOR(s)
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   << UPDATE
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | parser        | parsing flag 40 type 01 (origin) len 01 payload  (2) 02
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | parser        | parsing flag 40 type 02 (as-path) len 16 payload  (54) 0205 0000 FFDC 0000 FE4E 0000 FE4C 0000 A2DA 0000 FC00
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | parser        | parsing flag 40 type 03 (next-hop) len 04 payload  (9) 0A8C 0001
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | parser        | no withdrawn NLRI
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | parser        | parsing 1/1 nlri payload  (2) 00
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | parser        | parsed announce nlri 0.0.0.0/0 next-hop 10.140.0.1 payload  (2) 00
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | routes        | peer 10.140.0.1 ASN 65500   << UPDATE (1) (71)  attributes origin incomplete as-path [ 65500 65102 65100 41690 64512 ]
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | routes        | peer 10.140.0.1 ASN 65500   << UPDATE (1) nlri  (29) 0.0.0.0/0 next-hop 10.140.0.1
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   << UPDATE
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | parser        | parsing flag 40 type 01 (origin) len 01 payload  (2) 00
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | parser        | parsing flag 40 type 02 (as-path) len 0e payload  (34) 0203 0000 FFDC 0000 FE4E 0000 FE4C
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | parser        | parsing flag 40 type 03 (next-hop) len 04 payload  (9) 0A8C 0001
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | parser        | no withdrawn NLRI
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | parser        | parsing 1/1 nlri payload  (4) 080A
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | parser        | parsed announce nlri 10.0.0.0/8 next-hop 10.140.0.1 payload  (4) 080A
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | routes        | peer 10.140.0.1 ASN 65500   << UPDATE (2) (52)  attributes origin igp as-path [ 65500 65102 65100 ]
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | routes        | peer 10.140.0.1 ASN 65500   << UPDATE (2) nlri  (30) 10.0.0.0/8 next-hop 10.140.0.1
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   << UPDATE
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | parser        | no withdrawn NLRI
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | parser        | parsing 1/1 nlri payload  (14) 0CAC 1010 C0A8
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | parser        | parsed announce nlri 172.16.0.0/12 next-hop 10.140.0.1 payload  (7) 0CAC 10
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | parser        | parsing 1/1 nlri payload  (7) 10C0 A8
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | parser        | parsed announce nlri 192.168.0.0/16 next-hop 10.140.0.1 payload  (7) 10C0 A8
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | routes        | peer 10.140.0.1 ASN 65500   << UPDATE (3) (52)  attributes origin igp as-path [ 65500 65102 65100 ]
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | routes        | peer 10.140.0.1 ASN 65500   << UPDATE (3) nlri  (33) 172.16.0.0/12 next-hop 10.140.0.1
Sat, 16 May 2015 18:45:46 | INFO     | 17566  | routes        | peer 10.140.0.1 ASN 65500   << UPDATE (3) nlri  (34) 192.168.0.0/16 next-hop 10.140.0.1
Sat, 16 May 2015 18:45:48 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   << KEEPALIVE
Sat, 16 May 2015 18:45:48 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   >> KEEPALIVE
Sat, 16 May 2015 18:45:49 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   << KEEPALIVE
Sat, 16 May 2015 18:45:50 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   >> KEEPALIVE
Sat, 16 May 2015 18:45:51 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   << KEEPALIVE
Sat, 16 May 2015 18:45:52 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   >> KEEPALIVE
Sat, 16 May 2015 18:45:53 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   << KEEPALIVE
Sat, 16 May 2015 18:45:54 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   >> KEEPALIVE
^C
Sat, 16 May 2015 18:45:54 | INFO     | 17566  | reactor       | ^C received
Sat, 16 May 2015 18:45:54 | INFO     | 17566  | reactor       | Performing shutdown
Sat, 16 May 2015 18:45:54 | INFO     | 17566  | message       | Peer      10.140.0.1 ASN 65500   >> NOTIFICATION (6,3,"Peer De-configured")
Sat, 16 May 2015 18:45:55 | INFO     | 17566  | processes     | Terminating process mkdb

$ ls -al /tmp/all.db*
-rw-r----- 1 pyke pyke 50 mai   16 18:45 /tmp/all.db
-rw-r----- 1 pyke pyke 50 mai   16 18:44 /tmp/all.db.1
-rw-r----- 1 pyke pyke 50 mai   16 18:44 /tmp/all.db.2
-rw-r----- 1 pyke pyke 50 mai   16 18:44 /tmp/all.db.3

$ cat /tmp/all.db
0.0.0.0/0
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16

$ cat exabgp.conf
group peers {
   process mkdb {
      encoder json;
      peer-updates;
      neighbor-changes;
      receive-routes;
      run /home/pyke/work/exabgp-helpers/exaprefixdb/exaprefixdb.py /home/pyke/work/exabgp-helpers/exaprefixdb/exaprefixdb.conf;
   }
   neighbor 10.140.0.1 {
      router-id 192.168.37.128;
      local-address 192.168.37.128;
      local-as 65501;
      peer-as 65500;
      family {
         inet4 unicast;
      }
   }
}

$ cat exaprefixdb.conf
{
   "interval": 10,
   "exports":
   [
      { "database": "all" }
   ]
}

Tested with ExaBGP v3.4.10:

$ exabgp --version
ExaBGP : 3.4.10
Python : 2.7.6 (default, Mar 22 2014, 22:59:56)  [GCC 4.8.2]
Uname  : #79-Ubuntu SMP Tue Mar 10 20:06:50 UTC 2015

victor-sudakov · 2015-05-16T17:17:05Z

Yes to your first three questions. Because the file /var/tmp/prefixdb/all.db is indeed being created, but it contains only a linefeed and no prefixes. It would not be created if there were permission or starting problems.

As to the fourth question. The cisco router thinks it's sending prefixes to exabgp, but I cannot make sure it is receiving them.

victor-sudakov · 2015-05-16T17:22:15Z

And where did you learn about exabgp_log_all ? It's not found anywhere in the exabgp code as "pkg info -l exabgp | xargs grep -i exabgp_log_all" shows nothing.

Neither does "exabgp /usr/local/etc/exabgp/exabgp.conf" start anything in the foreground. In FreeBSD exabgp is started from a rc.d script and looks like this in the ps output:

97002 ?? R 0:06,39 /usr/local/bin/python2.7 /usr/local/lib/python2.7/site-packages/exabgp/application/bgp.py --folder /usr/local/etc/exabgp /usr/local/etc/exabgp/exabgp.conf
97022 ?? RL 0:00,01 /usr/local/bin/python2.7 /usr/local/etc/exabgp/exaprefixdb.py /usr/local/etc/exabgp/exaprefixdb.conf

pyke369 · 2015-05-16T17:57:19Z

"exabgp_log_all=true" is just a way of setting an environment variable before starting the exabgp process itself. you can find about environment variables by just starting exabgp with no argument (see option 4) to set env. variables below).

ExaBGP will automatically look for its configuration file (in windows ini format):
 * in the etc/exabgp folder located within the extracted tar.gz
 * in /etc/exabgp/exabgp.env

Individual configuration options can be set using environment variables, such as:
 > env exabgp.daemon.daemonize=true ./sbin/exabgp
or:
 > env exabgp.daemon.daemonize=true ./sbin/exabgp
or:
 > export exabgp.daemon.daemonize=true; ./sbin/exabgp

Multiple environment values can be set, the order of preference being:
 1) command line environment value using dot separated notation
 2) exported value from the shell using dot separated notation
 3) command line environment value using underscore separated notation
 4) exported value from the shell using underscore separated notation
 5) the value in the ini configuration file
 6) the built-in defaults

I don't know about the specifics of the FreeBSD packaging (I'm using Linux), but you should probably stop the stock ExaBGP daemon first (using the rc.d script, something like "/etc/rc.d/exabgp stop") and then try:

exabgp_daemon_daemonize=false exabgp_log_all=true exabgp <your_conf_file>

(to make sure ExaBGP does not daemonize itself). As for your trace above, I don't see any "routes" messages showing routes updates from the Cisco router. Try adding "exabgp_log_routes=true" just in case (but "exabgp_log_all=true" should already take care of this).

victor-sudakov · 2015-05-16T18:07:17Z

I don't see the environment variable "exabgp_log_all" anywhere in the exabgp code. How does it make sense?

thomas-mangin · 2015-05-16T19:09:56Z

./sbin/exabgp -h

https://github.com/Exa-Networks/exabgp/blob/master/lib/exabgp/configuration/setup.py
https://github.com/Exa-Networks/exabgp/blob/master/lib/exabgp/configuration/environment.py

victor-sudakov · 2015-05-17T06:19:28Z

It must be some FreeBSD specific issue then. Look:

$ /usr/local/sbin/exabgp
===> To start exabgp, you should use:
===> service exabgp start
$ /usr/local/sbin/exabgp -h
$ grep log_all /usr/local/lib/python2.7/site-packages/exabgp/configuration/environment.py
$

Is there anything I can put into exabgp.conf itself to enable debug?

pyke369 · 2015-05-17T08:49:21Z

Not in exabgp.conf directly, but in exabgp.env. This file should be located in the /etc/exabgp folder (and if it does not already exists, you can still use "exabgp --fi" to generate a first template and go from there).

[exabgp.log]
all = true
routes = true

ExaBGP will automatically look for its configuration file (in windows ini format):
 * in the etc/exabgp folder located within the extracted tar.gz
 * in /etc/exabgp/exabgp.env

victor-sudakov · 2015-05-17T10:42:39Z

Bingo, now I have a very detailed exabgp log with lots of updates. For some reason however, those updates are not passed to the script which I run from the "process" section. The only thing I see passed to this script is the line

{ "exabgp": "3.4.8", "time": 1431858703, "host" : "noc.sibptus.ru", "pid" : "3455", "ppid" : "1", "counter": 1, "type": "notification", "notification": "shutdown"}

and then nothing.

The exabgp log can be downloaded from ftp://ftp.sibptus.ru/pub/vas/exabgp.log.zip

It's mostly a question to Thomas I guess.

thomas-mangin · 2015-05-17T11:07:29Z

If you are running master make sure "parsed" is in your receive section.

victor-sudakov · 2015-05-17T11:18:16Z

@thomas-mangin I have published the complete log at ftp://ftp.sibptus.ru/pub/vas/exabgp.log.zip
it includes also the config file, could you please look if anything is missing or in the wrong section?

thomas-mangin · 2015-05-17T11:30:00Z

Updates generated from ExaBGP itself are not send to the script. If you need this feature you need to use master and then to change the section from "receive" to "send" to see what was generated.

victor-sudakov · 2015-05-17T11:51:33Z

@thomas-mangin I don't need updates generated by ExaBGP itself. I need updates received by ExaBGP from its two peers to be sent to the script.

Let's consider the prefix 31.211.40.0/22 received by ExaBGP from 212.73.125.216
You can see in the log that ExaBGP is receiving this prefix.

Sun, 17 May 2015 17:04:26 | INFO | 49814 | parser | parsed announce nlri 31.211.40.0/22 next-hop 213.183.111.225 payload 161F D328

Why is this prefix not sent to the script?

thomas-mangin · 2015-05-17T11:53:07Z

Because the current parser is brain dead, put the process section before your neighbors and tell me if it fixes it.

victor-sudakov · 2015-05-17T12:03:08Z

Yes, putting the process section before the neighbors fixes it. Who might think!

victor-sudakov · 2015-05-17T12:13:59Z

Now exaprefixdb is also working as expected. Thank you guys!

thomas-mangin · 2015-05-17T12:17:37Z

I will close this thread. Clearly no documentation was written but it is of little use for everyone and docs are a know issue - they do not need an entry here 😢

thomas-mangin added the feature-request label May 15, 2015

thomas-mangin closed this as completed May 17, 2015

acoul mentioned this issue Oct 5, 2017

exabgp-4.x support dpiekacz/gixlg#12

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Need offical documentation and script examples for collecting BGP routes #262

Need offical documentation and script examples for collecting BGP routes #262

victor-sudakov commented May 15, 2015

pyke369 commented May 15, 2015

thomas-mangin commented May 16, 2015

victor-sudakov commented May 16, 2015

pyke369 commented May 16, 2015

pyke369 commented May 16, 2015

victor-sudakov commented May 16, 2015

pyke369 commented May 16, 2015

victor-sudakov commented May 16, 2015

victor-sudakov commented May 16, 2015

pyke369 commented May 16, 2015

victor-sudakov commented May 16, 2015

thomas-mangin commented May 16, 2015

victor-sudakov commented May 17, 2015

pyke369 commented May 17, 2015

victor-sudakov commented May 17, 2015

thomas-mangin commented May 17, 2015

victor-sudakov commented May 17, 2015

thomas-mangin commented May 17, 2015

victor-sudakov commented May 17, 2015

thomas-mangin commented May 17, 2015

victor-sudakov commented May 17, 2015

victor-sudakov commented May 17, 2015

thomas-mangin commented May 17, 2015

Need offical documentation and script examples for collecting BGP routes #262

Need offical documentation and script examples for collecting BGP routes #262

Comments

victor-sudakov commented May 15, 2015

pyke369 commented May 15, 2015

thomas-mangin commented May 16, 2015

victor-sudakov commented May 16, 2015

pyke369 commented May 16, 2015

pyke369 commented May 16, 2015

victor-sudakov commented May 16, 2015

pyke369 commented May 16, 2015

victor-sudakov commented May 16, 2015

victor-sudakov commented May 16, 2015

pyke369 commented May 16, 2015

victor-sudakov commented May 16, 2015

thomas-mangin commented May 16, 2015

victor-sudakov commented May 17, 2015

pyke369 commented May 17, 2015

victor-sudakov commented May 17, 2015

thomas-mangin commented May 17, 2015

victor-sudakov commented May 17, 2015

thomas-mangin commented May 17, 2015

victor-sudakov commented May 17, 2015

thomas-mangin commented May 17, 2015

victor-sudakov commented May 17, 2015

victor-sudakov commented May 17, 2015

thomas-mangin commented May 17, 2015