Added pywbemlistener command

[andy-maier]

At this point, the new 'pywbemlistener' command is ready for review and merge.
For details, see commit message.

DEFERRED to separate issues:

• Add tests - issue Tests for pywbemlistener command #981
• Add documentation - issue Finish the documentation of pywbemlistener #980

DROPPED:

• Rename those of the PYWBEMCLI_* environment variables that are common. It seems that for the same reason we keep the CERTFILE/KEYFILE variables separate, we should also keep the other two variables for PDB and LOG separate.

[coveralls]

Coverage decreased (-11.1%) to 81.719% when pulling 3850827 on andy/pywbemlistener-command into c0b4b35 on master.

[andy-maier]

DISCUSSION-DONE: How to specify the indication processing options

Proposal:

• Separate options that can all be specified -> each indication can have multiple processors
• --indi-file FILE - append to FILE, in a one-line format
• --indi-display - display on stdout, in a one-line format
• --indi-call MODULE.FUNC - call a Python function
• --indi-format FORMAT - define one-line format using new string formatting

The latest update of the PR implements this proposal.

Update: We decided to go with this proposal for now and adjust as we learn.

[andy-maier]

DISCUSSION: How to specify logging

There are some differences to pywbemcli: In pywbemcli, logging is basically deferred to pywbem which makes sense for that command. One pywbemcli invocation deals with one WBEM server, and each invocation can specify a separate log file if needed. In case of pywbemlistener, there are multiple listeners running, each of which should log to a separate file. Another difference is that in pywbemlistener we want to simply log stdout/stderr of the run commands and not internal APIs or HTTP protocol (even though that could be added at some point).

Proposal:

• Instead of using the logging specification from pywbemcli, use a simple --logdir option that specifies the path name of a log directory. The file name is automatically generated from the listener name and is pywbemlistener_NAME.log in that directory. Not specifying the --logdir option disables logging.

The latest update of the PR implements this proposal.

COMMENT:KS I have no problems with this proposal.

[KSchopmeyer]

I did some review and made some comments. I am testing with open pegasus but getting inconsistent results from pegasus test indcation provider for some reason. I have gotten display in some cases but not all.

[KSchopmeyer]

I thin a flag that would provide summary results either to a running listener or to one closing (number received, starttim, end time,)
would be logical. Since we cannot sync start / end of listener against start/end of sending indications from server anything like indications/sec may well be meaningless unless presented as a table (i.e per some period) but would be nice

I am going to continue to test with openpegasus to sort out why I am having issues getting indications on consistent basis.

[KSchopmeyer]

Some test results from pegasus

===================================

pywbemlistener run lis1 --port 50000 --protocol http --indi-display --indi-format "{dt} {c}"
Running listener lis1 at http://localhost:50000
2021-05-15 10:11:56.323227-05:00 Test_IndicationProviderClass
2021-05-15 10:11:56.326560-05:00 Test_IndicationProviderClass
...

==================================================

$ pywbemlistener run lis1 --port 50000 --protocol http --indi-display
Running listener lis1 at http://localhost:50000
2021-05-15 10:13:12.414368-05:00 127.0.0.1 Test_IndicationProviderClass IndicationIdentifier='41' IndicationTime=CIMDateTime(cimtype='datetime', datetime=datetime.datetime(2021, 5, 15, 10, 13, 12, 406809, tzinfo=MinutesFromUTC(offset=-300)), timedelta=None, precision=None), minutes_from_utc=-300) CorrelatedIndications=[] MethodName='SendTestIndicationsCount' SequenceContext='PG:IndicationService-PG:127-0-1-1-1621091383113453-1621091592338226' SequenceNumber=Sint64(cimtype='sint64', minvalue=-9223372036854775808, maxvalue=9223372036854775807, 0)
2021-05-15 10:13:12.418567-05:00 127.0.0.1 Test_IndicationProviderClass IndicationIdentifier='42' IndicationTime=CIMDateTime(cimtype='datetime', datetime=datetime.datetime(2021, 5, 15, 10, 13, 12, 408907, tzinfo=MinutesFromUTC(offset=-300)), timedelta=None, precision=None), minutes_from_utc=-300) CorrelatedIndications=[] MethodName='SendTestIndicationsCount' SequenceContext='PG:IndicationService-PG:127-0-1-1-1621091383113453-1621091592338226' SequenceNumber=Sint64(cimtype='sint64', minvalue=-9223372036854775808, maxvalue=9223372036854775807, 1)

. . .

============================================

pywbemlistener run lis1 --port 50000 --protocol http --indi-display --indi-format "{dt} {h} {p}"
Running listener lis1 at http://localhost:50000
2021-05-15 10:16:52.945235-05:00 127.0.0.1 IndicationIdentifier='61' IndicationTime=CIMDateTime(cimtype='datetime', datetime=datetime.datetime(2021, 5, 15, 10, 16, 52, 938319, tzinfo=MinutesFromUTC(offset=-300)), timedelta=None, precision=None), minutes_from_utc=-300) CorrelatedIndications=[] MethodName='SendTestIndicationsCount' SequenceContext='PG:IndicationService-PG:127-0-1-1-1621091383113453-1621091812873408' SequenceNumber=Sint64(cimtype='sint64', minvalue=-9223372036854775808, maxvalue=9223372036854775807, 0)
2021-05-15 10:16:52.949150-05:00 127.0.0.1 IndicationIdentifier='62' IndicationTime=CIMDateTime(cimtype='datetime', datetime=datetime.datetime(2021, 5, 15, 10, 16, 52, 940260, tzinfo=MinutesFromUTC(offset=-300)), timedelta=None, precision=None), minutes_from_utc=-300) CorrelatedIndications=[] MethodName='SendTestIndicationsCount' SequenceContext='PG:IndicationService-PG:127-0-1-1-1621091383113453-1621091812873408' SequenceNumber=Sint64(cimtype='sint64', minvalue=-9223372036854775808, maxvalue=9223372036854775807, 1)

==============================================

Test with indi-call where i created a new package with one very simple function

def myindication(indication, host):
    print(indication.tomof())

Produces

(pycli39issue395) kschopmeyer@leonard:~/pywbem/pycli39issue395/pywbemtools/testlistfun(andy/pywbemlistener-command)$ pywbemlistener run lis1 --port 50000 --protocol http --indi-display --indi-call testlistfun.myindication
Running listener lis1 at http://localhost:50000
instance of Test_IndicationProviderClass {
   IndicationIdentifier = "1";
   IndicationTime = "20210515103810.732697-300";
   CorrelatedIndications = { };
   MethodName = "SendTestIndicationsCount";
   SequenceContext =
      "PG:IndicationService-PG:127-0-1-1-1621091383113453-1621093090648398";
   SequenceNumber = 0;
};

2021-05-15 10:38:10.740901-05:00 127.0.0.1 Test_IndicationProviderClass IndicationIdentifier='1' IndicationTime=CIMDateTime(cimtype='datetime', datetime=datetime.datetime(2021, 5, 15, 10, 38, 10, 732697, tzinfo=MinutesFromUTC(offset=-300)), timedelta=None, precision=None), minutes_from_utc=-300) CorrelatedIndications=[] MethodName='SendTestIndicationsCount' SequenceContext='PG:IndicationService-PG:127-0-1-1-1621091383113453-1621093090648398' SequenceNumber=Sint64(cimtype='sint64', minvalue=-9223372036854775808, maxvalue=9223372036854775807, 0)
instance of Test_IndicationProviderClass {
   IndicationIdentifier = "2";
   IndicationTime = "20210515103810.734972-300";
   CorrelatedIndications = { };
   MethodName = "SendTestIndicationsCount";
   SequenceContext =
      "PG:IndicationService-PG:127-0-1-1-1621091383113453-1621093090648398";
   SequenceNumber = 1;
};

2021-05-15 10:38:10.745286-05:00 127.0.0.1 Test_IndicationProviderClass IndicationIdentifier='2' IndicationTime=CIMDateTime(cimtype='datetime', datetime=datetime.datetime(2021, 5, 15, 10, 38, 10, 734972, tzinfo=MinutesFromUTC(offset=-300)), timedelta=None, precision=None), minutes_from_utc=-300) CorrelatedIndications=[] MethodName='SendTestIndicationsCount' SequenceContext='PG:IndicationService-PG:127-0-1-1-1621091383113453-1621093090648398' SequenceNumber=Sint64(cimtype='sint64', minvalue=-9223372036854775808, maxvalue=9223372036854775807, 1)

==============================================

[KSchopmeyer]

did first test of the MODULE.FUNCTION capability shown above but to do it I created a temp package. I did not figure out how to defined a module alone. (I assumed it would be a python file at the directory level of the call but that did not work).

When I execute sys.path in python 3.9, the current directory shows up in the list. ..'/home/kschopmeyer/pywbem/pycli39issue395/pywbemtools']

[andy-maier]

On using --indi-call: I tried both a single-file module and a package to verify that both work:

Single-file module:

./tmp_display_callback.py - with function display()

$ pywbemlistener start lis1 --protocol http --indi-call tmp_display_callback.display

Package:

./tmp_try/__init__.py - empty
./tmp_try/display_callback.py - with function display()

$ pywbemlistener start lis1 --protocol http --indi-call tmp_try.display_callback.display

Both approaches require that Python includes the current directory in its module search path. I have verified that sys.path contains the current directory at the end, on my macOS, in a virtualenv created with virtualenv:

'/Users/maiera/virtualenvs/pywbemtools38/bin'
'/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python38.zip'
'/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8'
'/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload'
'/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages'
'/Users/maiera/PycharmProjects/pywbem/pywbemtools' <-- current directory

DISCUSSION:

Can you please verify whether the current directory is in your sys.path? BTW, strangely enough, invoking just the python command in that same environment in addition has the empty path as the first item in the list, which also is the current directory.

The latest update to this PR adds the current directory to the module search path before we import the module with the callback function.

NOTE: Tested on latest version over weekend and it works. Can use module in current directory or a package.

[KSchopmeyer]

Note that a number of the comments are ideas. Thus, if we can do the count through the function call, that would be fine with me for now. Most important. I thin that functionally this is very close to merge and merging it is more important that more functionality right now. That way I can work with it while you are gone.

[KSchopmeyer]

NOTE: I have tested most of the functionality using OpenPegasus as indication generator. But this is only with http. Have not tested https at all but I can do that after this commit. We do have an issue between the OpenPegasus test indication and listener when I send significant groups of indications, ex. 100 at a time. Works a few times and then slows down and/or stops. I do not know which side is the issue. Note that the first group passes VERY fast (within a couple of seconds). I suspect OpenPegasus but it is going to take more detailed testing to sort that out.

[andy-maier]

We agreed in the meeting to merge this PR with the changes as documented.