Internal Server Error loading prefixes.csv

[sraustein]

prefixes.csv is
{{{
test 0.0.0.0/0
}}}

i try to load it

{{{
Internal Server Error
}}}

Trac ticket #773 component gui priority critical, owner melkins, created by randy on 2015-08-07T04:04:38Z, last modified 2016-04-22T16:07:21Z

[sraustein]

Logs

Trac comment by sra on 2015-08-07T04:14:44Z

[sraustein]

nothing about test.csv in /var/log/messages

not sure how to get gooey to log

Trac comment by randy on 2015-08-07T04:29:35Z

[sraustein]

Gotta love this one:

{{{
[Fri Aug 07 04:00:28.903301 2015] [:error] [pid 10857:tid 2980014912] ERROR 2015-08-07 04:00:28,890 django.request Internal Server Error: /rpki/conf/import_prefixes
[Fri Aug 07 04:00:28.903339 2015] [:error] [pid 10857:tid 2980014912] Traceback (most recent call last):
[Fri Aug 07 04:00:28.903348 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 112, in get_response
[Fri Aug 07 04:00:28.903355 2015] [:error] [pid 10857:tid 2980014912] response = wrapped_callback(request, _callback_args, *_callback_kwargs)
[Fri Aug 07 04:00:28.903363 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view
[Fri Aug 07 04:00:28.903370 2015] [:error] [pid 10857:tid 2980014912] return view_func(request, _args, *_kwargs)
[Fri Aug 07 04:00:28.903376 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/rpki/gui/decorators.py", line 28, in _tls_required
[Fri Aug 07 04:00:28.903383 2015] [:error] [pid 10857:tid 2980014912] return f(request, _args, *_kwargs)
[Fri Aug 07 04:00:28.903390 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/rpki/gui/app/views.py", line 106, in wrapped_fn
[Fri Aug 07 04:00:28.903397 2015] [:error] [pid 10857:tid 2980014912] return f(request, _args, *_kwargs)
[Fri Aug 07 04:00:28.903404 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/rpki/gui/app/views.py", line 391, in import_prefixes
[Fri Aug 07 04:00:28.903411 2015] [:error] [pid 10857:tid 2980014912] ignore_missing_children=form.cleaned_data['ignore_missing_children']
[Fri Aug 07 04:00:28.903418 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 399, in inner
[Fri Aug 07 04:00:28.903433 2015] [:error] [pid 10857:tid 2980014912] return func(_args, *_kwargs)
[Fri Aug 07 04:00:28.903440 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/rpki/irdb/zookeeper.py", line 925, in load_prefixes
[Fri Aug 07 04:00:28.903446 2015] [:error] [pid 10857:tid 2980014912] child = self.resource_ca.children.get(handle = handle)
[Fri Aug 07 04:00:28.903453 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/django/db/models/manager.py", line 151, in get
[Fri Aug 07 04:00:28.903459 2015] [:error] [pid 10857:tid 2980014912] return self.get_queryset().get(_args, *_kwargs)
[Fri Aug 07 04:00:28.903465 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 301, in get
[Fri Aug 07 04:00:28.903472 2015] [:error] [pid 10857:tid 2980014912] num = len(clone)
[Fri Aug 07 04:00:28.903478 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 77, in len
[Fri Aug 07 04:00:28.903484 2015] [:error] [pid 10857:tid 2980014912] self._fetch_all()
[Fri Aug 07 04:00:28.903490 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 854, in _fetch_all
[Fri Aug 07 04:00:28.903496 2015] [:error] [pid 10857:tid 2980014912] self._result_cache = list(self.iterator())
[Fri Aug 07 04:00:28.903503 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 220, in iterator
[Fri Aug 07 04:00:28.903509 2015] [:error] [pid 10857:tid 2980014912] for row in compiler.results_iter():
[Fri Aug 07 04:00:28.903515 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 710, in results_iter
[Fri Aug 07 04:00:28.903521 2015] [:error] [pid 10857:tid 2980014912] for rows in self.execute_sql(MULTI):
[Fri Aug 07 04:00:28.903527 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 781, in execute_sql
[Fri Aug 07 04:00:28.903534 2015] [:error] [pid 10857:tid 2980014912] cursor.execute(sql, params)
[Fri Aug 07 04:00:28.903540 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
[Fri Aug 07 04:00:28.903546 2015] [:error] [pid 10857:tid 2980014912] return self.cursor.execute(sql, params)
[Fri Aug 07 04:00:28.903552 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/django/db/utils.py", line 99, in exit
[Fri Aug 07 04:00:28.903558 2015] [:error] [pid 10857:tid 2980014912] six.reraise(dj_exc_type, dj_exc_value, traceback)
[Fri Aug 07 04:00:28.903565 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
[Fri Aug 07 04:00:28.903571 2015] [:error] [pid 10857:tid 2980014912] return self.cursor.execute(sql, params)
[Fri Aug 07 04:00:28.903577 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 124, in execute
[Fri Aug 07 04:00:28.903583 2015] [:error] [pid 10857:tid 2980014912] return self.cursor.execute(query, args)
[Fri Aug 07 04:00:28.903589 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
[Fri Aug 07 04:00:28.903596 2015] [:error] [pid 10857:tid 2980014912] self.errorhandler(self, exc, value)
[Fri Aug 07 04:00:28.903602 2015] [:error] [pid 10857:tid 2980014912] File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
[Fri Aug 07 04:00:28.903608 2015] [:error] [pid 10857:tid 2980014912] raise errorclass, errorvalue
[Fri Aug 07 04:00:28.903614 2015] [:error] [pid 10857:tid 2980014912] OperationalError: (1267, "Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '='")
}}}

Trac comment by sra on 2015-08-07T04:30:51Z

[sraustein]

GUI mostly throws its logs at Apache, so on Debian and Ubuntu it ends up in /var/log/apache2/

Trac comment by sra on 2015-08-07T04:32:28Z

[sraustein]

/var/log/apache2/

thanks

Trac comment by randy on 2015-08-07T04:36:44Z

[sraustein]

my favorite bit
{{{
(latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '='")
}}}

bork bork

so how to move forward? let TestRoot own all the roas for now?

randy

Trac comment by randy on 2015-08-07T04:39:44Z

[sraustein]

i used rpkic to

{{{
rpkic> load_prefixes /usr/share/rpki/test.prefixes.csv
}}}

containing
{{{
TestRoot 0.0.0.0/0
TestRoot 0::/0
test 0.0.0.0/0
}}}

and now gooey blows chunks on login

{{{
[Fri Aug 07 04:47:24.633741 2015] [:error] [pid 10858:tid 2963229504] OverflowError: long too big to convert
[Fri Aug 07 04:47:40.943682 2015] [:error] [pid 10858:tid 2870909760] ERROR 2015-08-07 04:47:40,943 django.request Internal Server Error: /rpki/
[Fri Aug 07 04:47:40.943707 2015] [:error] [pid 10858:tid 2870909760] Traceback (most recent call last):
[Fri Aug 07 04:47:40.943713 2015] [:error] [pid 10858:tid 2870909760] File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 112, in get_response
[Fri Aug 07 04:47:40.943718 2015] [:error] [pid 10858:tid 2870909760] response = wrapped_callback(request, _callback_args, *_callback_kwargs)
[Fri Aug 07 04:47:40.943724 2015] [:error] [pid 10858:tid 2870909760] File "/usr/lib/python2.7/dist-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view
[Fri Aug 07 04:47:40.943729 2015] [:error] [pid 10858:tid 2870909760] return view_func(request, _args, *_kwargs)
[Fri Aug 07 04:47:40.943734 2015] [:error] [pid 10858:tid 2870909760] File "/usr/lib/python2.7/dist-packages/rpki/gui/decorators.py", line 28, in _tls_required
[Fri Aug 07 04:47:40.943739 2015] [:error] [pid 10858:tid 2870909760] return f(request, _args, *_kwargs)
[Fri Aug 07 04:47:40.943744 2015] [:error] [pid 10858:tid 2870909760] File "/usr/lib/python2.7/dist-packages/rpki/gui/app/views.py", line 106, in wrapped_fn
[Fri Aug 07 04:47:40.943748 2015] [:error] [pid 10858:tid 2870909760] return f(request, _args, *_kwargs)
[Fri Aug 07 04:47:40.943753 2015] [:error] [pid 10858:tid 2870909760] File "/usr/lib/python2.7/dist-packages/rpki/gui/app/views.py", line 229, in dashboard
[Fri Aug 07 04:47:40.943758 2015] [:error] [pid 10858:tid 2870909760] unused_prefixes = my_prefixes.difference(used_prefixes)
[Fri Aug 07 04:47:40.943763 2015] [:error] [pid 10858:tid 2870909760] File "/usr/lib/python2.7/dist-packages/rpki/gui/app/range_list.py", line 97, in difference
[Fri Aug 07 04:47:40.943769 2015] [:error] [pid 10858:tid 2870909760] xmin = cur.max + 1
[Fri Aug 07 04:47:40.943773 2015] [:error] [pid 10858:tid 2870909760] OverflowError: long too big to convert
}}}

Trac comment by randy on 2015-08-07T04:49:38Z

[sraustein]

so how to move forward? let TestRoot own all the roas for now?

Given that the character set war looks like an HTTP/Django thing, you might try using rpkic to load the CSV files instead of the GUI.

Other than that, I'm too fried to make intelligent suggestions at this point.

Trac comment by sra on 2015-08-07T04:51:15Z

[sraustein]

we'll stick with rpkic until melkins fixes gooey

Trac comment by randy on 2015-08-07T04:53:12Z

[sraustein]

Rob fixed this for MySQL by forcing the charset to latin-1.

Trac comment by melkins on 2016-04-22T16:07:21Z

[sraustein]

Closed with resolution fixed