Support loading plugins from multiple packages. #948
Conversation
Signed-off-by: David Venable <[email protected]>
|
I created a sample Data Prepper plugin which demonstrates this behavior: https://github.com/dlvenable/data-prepper-sample-plugin |
Codecov Report
@@ Coverage Diff @@
## main #948 +/- ##
============================================
- Coverage 91.40% 91.36% -0.04%
- Complexity 673 683 +10
============================================
Files 83 84 +1
Lines 1955 1993 +38
Branches 165 168 +3
============================================
+ Hits 1787 1821 +34
- Misses 129 132 +3
- Partials 39 40 +1
Continue to review full report at Codecov.
|
| pluginPropertiesInputStream = pluginPropertiesUrl.openStream(); | ||
| pluginProperties.load(pluginPropertiesInputStream); |
There was a problem hiding this comment.
Help me understand what is happening here a little better. So you add your package name to the end of META-INF/data-prepper.plugins.properties file under the org.opensearch.dataprepper.plugin.packages variable. Then here we are actually connecting to the package URL and downloading the properties of that package (like package name, and the distribution URL) so we know where to actually go to get the package?
There was a problem hiding this comment.
This is not downloading anything. Issue #321 discusses downloading jars.
As a developer, you can create your own plugin jar file. If you do, you will also include the META-INF/data-prepper.plugins.properties file and specify your plugin package. You can see that in my sample.
Then if you run Data Prepper with your plugin in the classpath, Data Prepper will read this file and be able to scan that package from the local jar file.
Now running Data Prepper with a jar in the classpath is a little tricky since we have the uber-jar. You can see how I did it in the sample. I expect that #305 will help with this in the future since Data Prepper could include all jars in a plugins directory.
| final InputStream pluginPropertiesInputStream; | ||
| try { | ||
| pluginPropertiesInputStream = pluginPropertiesUrl.openStream(); |
There was a problem hiding this comment.
Declaring and assigning pluginPropertiesInputStream can be a single line.
try {
final InputStream pluginPropertiesInputStream = pluginPropertiesUrl.openStream();
| final String packagesRawString = pluginProperties.getProperty("org.opensearch.dataprepper.plugin.packages"); | ||
|
|
||
| final String[] currentPackageNames = packagesRawString.split(","); |
There was a problem hiding this comment.
getProperty will return null if "org.opensearch.dataprepper.plugin.packages" does not exist. That would result in a NPE on line 83.
| } | ||
|
|
||
| if(packageNames.isEmpty()) { | ||
| packageNames.add(DEFAULT_PLUGINS_CLASSPATH); |
There was a problem hiding this comment.
Take it or leave it: I might suggest adding this warning statement here to be consistent with the messaging you are providing in getUniquePackageNames:
LOG.warn("Unable to load packages from data-prepper.plugins.properties file. Reverting to default plugin package.");
There was a problem hiding this comment.
Good suggestion! I took it.
Signed-off-by: David Venable <[email protected]>
Description
This PR allows Data Prepper to support plugins in packages other than
com.amazon.dataprepper.plugins.Data Prepper reads all resources (files embedded in jars) of
META-INF/data-prepper.plugins.properties. It then looks for a propertyorg.opensearch.dataprepper.plugin.packagesfrom each one. The value of this property is a comma-delimited list of packages.Data Prepper will scan all of these packages for plugins.
Note that because of the uber-jar, Data Prepper can only have one file for all packages. However, plugin authors can create their own jar files with this file. As long as it is on the classpath, their custom plugin will be available.
It may be worthwhile to move some plugins in future PRs.
Issues Resolved
#379
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.