-
Notifications
You must be signed in to change notification settings - Fork 217
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ENH: enable TLS/SSL through local filepath in http source #359
Changes from all commits
956caef
5756f6f
35aa770
f6f799c
e947f7f
763be20
d496795
d54fcf7
40a9181
100d657
0583fe9
103bd01
8611134
bbcf96e
151ca19
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
-----BEGIN CERTIFICATE----- | ||
MIICHTCCAYYCCQD4hqYeYDQZADANBgkqhkiG9w0BAQUFADBSMQswCQYDVQQGEwJV | ||
UzELMAkGA1UECAwCVFgxDzANBgNVBAcMBkF1c3RpbjEPMA0GA1UECgwGQW1hem9u | ||
MRQwEgYDVQQLDAtEYXRhcHJlcHBlcjAgFw0yMTA2MjUxOTIzMTBaGA8yMTIxMDYw | ||
MTE5MjMxMFowUjELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAlRYMQ8wDQYDVQQHDAZB | ||
dXN0aW4xDzANBgNVBAoMBkFtYXpvbjEUMBIGA1UECwwLRGF0YXByZXBwZXIwgZ8w | ||
DQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAKrb3YhdKbQ5PtLHall10iLZC9ZdDVrq | ||
HOvqVSM8NHlL8f82gJ3l0n9k7hYc5eKisutaS9eDTmJ+Dnn8xn/qPSKTIq9Wh+OZ | ||
O+e9YEEpI/G4F9KpGULgMyRg9sJK0GlZdEt9o5GJNJIJUkptJU5eiLuE0IV+jyJo | ||
Nvm8OE6EJPqxAgMBAAEwDQYJKoZIhvcNAQEFBQADgYEAjgnX5n/Tt7eo9uakIGAb | ||
uBhvYdR8JqKXqF9rjFJ/MIK7FdQSF/gCdjnvBhzLlZFK/Nb6MGKoSKm5Lcr75LgC | ||
FyhIwp3WlqQksiMFnOypYVY71vqDgj6UKdMaOBgthsYhngj8lC+wsVzWqQvkJ2Qg | ||
/GAIzJwiZfXiaevQHRk79qI= | ||
-----END CERTIFICATE----- |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
-----BEGIN RSA PRIVATE KEY----- | ||
MIICXAIBAAKBgQCq292IXSm0OT7Sx2pZddIi2QvWXQ1a6hzr6lUjPDR5S/H/NoCd | ||
5dJ/ZO4WHOXiorLrWkvXg05ifg55/MZ/6j0ikyKvVofjmTvnvWBBKSPxuBfSqRlC | ||
4DMkYPbCStBpWXRLfaORiTSSCVJKbSVOXoi7hNCFfo8iaDb5vDhOhCT6sQIDAQAB | ||
AoGANrrhFqpJDpr7vcb1ER0Fp/YArbT27zVo+EUC6puBb41dQlQyFOImcHpjLaAq | ||
H1PgnjU5cBp2hGQ+vOK0rwrYc/HNl6vfh6N3NbDptMiuoBafRJA9JzYourAM09BU | ||
zmXyr61Yn3KHzx1PRwWe37icX93oXP3P0qHb3dI1ZF4jG0ECQQDU5N/a7ogoz2zn | ||
ZssD6FvUOUQDsdBWdXmhUvg+YdZrV44e4xk+FVzwEONoRktEYKz9MFXlsgNHr445 | ||
KRguHWcJAkEAzXQkwOkN8WID1wrwoobUIMbZSGAZzofwkKXgTTnllnT1qOQXuRbS | ||
aCMejFEymBBef4aXP6N4+va2FKW/MF34aQJAO2oMl1sOoOUSrZngepy0VAwPUUCk | ||
thxe74jqQu6nGpn6zd/vQYZQw6bS8Fz90H1yic6dilcd1znFZWp0lxoZkQJBALeI | ||
xoBycRsuFQIYasi1q3AwUtBd0Q/3zkZZeBtk2hzjFMUwJaUZpxKSNOrialD/ZnuD | ||
jz+xWBTRKe0d98JMX+kCQCmsJEj/HYQAC1GamZ7JQWogRSRF2KTgTWRaDXDxy0d4 | ||
yUQgwHB+HZLFcbi1JEK6eIixCsX8iifrrkteh+1npJ0= | ||
-----END RSA PRIVATE KEY----- |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
-----BEGIN ENCRYPTED PRIVATE KEY----- | ||
MIICojAcBgoqhkiG9w0BDAEDMA4ECAd2FKZw2oGwAgIIAASCAoDTgiaXkazaotc7 | ||
SxQK3bX34sEvdkLXg/O4ZpHTb0f4gLPxNhDe7ZPKrAS2TdywpSHT0189MVl+PIvw | ||
4YQDaGVHL1SM5ukJu+PQkfQAMigdCJ+bUsG6hkrUDC74qYhHZHj1yVGavL6I4KHT | ||
Ixh9IV2GMRS4m6HGJ17nYsdiTFFNdK++WcTMxbVWv3SNdKGZG79T32pjMxuIUPWr | ||
3dB+ZXM+FSqwlBLZxPvvjlP6ETw7QXrlBHcQh1tHSh10bM+q0c5CktZoXLwpc+gv | ||
ZsGXzpVjqFrAw4Vw0ikJl1mUCoGOtqqP0P6QWwbIJZBxNoO0MvWcXW+U3AGNFFze | ||
nMR8UTXdga2l1Lx7pokQkWUpp48SDRjDx/RdZTRXCgtRcKuBcm0x2lxNILdwOzjJ | ||
5GlKMvvc2OXXTnYqSCTqdfbuR3XBYmWgFki92D6JnVIYq+QJr5qj8IJDJ7mADQ1i | ||
Za6PEJnrT641fLeSKRq7QiTydMQ3JXa9DFqUPwdZPPHLr/hC19sWHrq7gxvhkcLI | ||
wrTtTIi8/iV4IVaiHk7YU83IM6sGExabQ3BRXcHMr+7i1vVxtEsFNC6ycTfJ8gpJ | ||
YsnpXUQe912l5sk7GRSP1InNRF7kzMD0QeOAQ0UVfmsbHOPSXvD7fXkJWIb6N+zW | ||
qCQYRmBwc7Bz2KZein5MLsMeNz2AWj/DcA2fMC+4+QtI0nF5BFtaR0V0npWhsbPu | ||
3rj+AXipnvVhDIkfl8495j7ybCBj7HAZk8Ux8GmiZ3PGFO1C7XCQaLPWJ4Aw4Kb3 | ||
EUqtVtpbwsCov5PDmMDXgz8qOxWMdQsP/dPF1HnVAg7SoFG9xA4nHdZ2WAFZwYtf | ||
rRxEd7br | ||
-----END ENCRYPTED PRIVATE KEY----- |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5,7 +5,9 @@ plugins { | |
dependencies { | ||
implementation project(':data-prepper-api') | ||
implementation project(':data-prepper-plugins:blocking-buffer') | ||
implementation project(':data-prepper-plugins:common') | ||
implementation "com.linecorp.armeria:armeria:1.9.2" | ||
implementation "commons-io:commons-io:2.11.0" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We ought to move the versions of dependencies we use in multiple places out to the external file with the others that we've already defined. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Right now we have a versionsMap in our Gradle 7 should help us improve this, but it is not possible to update presently due to the OpenSearch plugin. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We could add commons-io version into versionMap. There might be other common packages. We should do that in a separate PR. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please make a separate PR for this. |
||
testImplementation 'org.assertj:assertj-core:3.20.2' | ||
testImplementation "org.hamcrest:hamcrest:2.2" | ||
} | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -17,12 +17,17 @@ | |
import com.amazon.dataprepper.model.configuration.PluginSetting; | ||
import com.amazon.dataprepper.model.record.Record; | ||
import com.amazon.dataprepper.model.source.Source; | ||
import com.amazon.dataprepper.plugins.certificate.CertificateProvider; | ||
import com.amazon.dataprepper.plugins.certificate.model.Certificate; | ||
import com.amazon.dataprepper.plugins.source.loghttp.certificate.CertificateProviderFactory; | ||
import com.linecorp.armeria.server.Server; | ||
import com.linecorp.armeria.server.ServerBuilder; | ||
import com.linecorp.armeria.server.throttling.ThrottlingService; | ||
import org.slf4j.Logger; | ||
import org.slf4j.LoggerFactory; | ||
|
||
import java.io.ByteArrayInputStream; | ||
import java.nio.charset.StandardCharsets; | ||
import java.time.Duration; | ||
import java.util.concurrent.ExecutionException; | ||
import java.util.concurrent.ScheduledThreadPoolExecutor; | ||
|
@@ -32,10 +37,12 @@ public class HTTPSource implements Source<Record<String>> { | |
private static final Logger LOG = LoggerFactory.getLogger(HTTPSource.class); | ||
|
||
private final HTTPSourceConfig sourceConfig; | ||
private final CertificateProviderFactory certificateProviderFactory; | ||
private Server server; | ||
|
||
public HTTPSource(final PluginSetting pluginSetting) { | ||
sourceConfig = HTTPSourceConfig.buildConfig(pluginSetting); | ||
certificateProviderFactory = new CertificateProviderFactory(sourceConfig); | ||
} | ||
|
||
@Override | ||
|
@@ -45,8 +52,19 @@ public void start(Buffer<Record<String>> buffer) { | |
} | ||
if (server == null) { | ||
final ServerBuilder sb = Server.builder(); | ||
// TODO: allow tls/ssl | ||
sb.http(sourceConfig.getPort()); | ||
if (sourceConfig.isSsl()) { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: do we want to have There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think isSSL makes more sense. This getter is autogenerated. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we should stick with |
||
LOG.info("SSL/TLS is enabled."); | ||
final CertificateProvider certificateProvider = certificateProviderFactory.getCertificateProvider(); | ||
final Certificate certificate = certificateProvider.getCertificate(); | ||
// TODO: enable encrypted key with password | ||
sb.https(sourceConfig.getPort()).tls( | ||
new ByteArrayInputStream(certificate.getCertificate().getBytes(StandardCharsets.UTF_8)), | ||
new ByteArrayInputStream(certificate.getPrivateKey().getBytes(StandardCharsets.UTF_8) | ||
) | ||
); | ||
} else { | ||
sb.http(sourceConfig.getPort()); | ||
} | ||
sb.maxNumConnections(sourceConfig.getMaxConnectionCount()); | ||
final int requestTimeoutInMillis = sourceConfig.getRequestTimeoutInMillis(); | ||
// Allow 2*requestTimeoutInMillis to accommodate non-blocking operations other than buffer writing. | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
/* | ||
* SPDX-License-Identifier: Apache-2.0 | ||
* | ||
* The OpenSearch Contributors require contributions made to | ||
* this file be licensed under the Apache-2.0 license or a | ||
* compatible open source license. | ||
* | ||
* Modifications Copyright OpenSearch Contributors. See | ||
* GitHub history for details. | ||
*/ | ||
|
||
package com.amazon.dataprepper.plugins.source.loghttp.certificate; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not a blocker for now, but do we want to centralize the SSL certificate logic so we don't have these CertificateProviderFactories all over the codebase? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The reason I did not centralize this class in the PR is b/c right now it takes the source plugin config as input arg. We will need to refactor out a common SSLConfig with all possible relevant parameters in order to do this centralization. The SSLConfig will be reused by the existing source plugins. I will reserve it for future iteration and open an issue. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
|
||
import com.amazon.dataprepper.plugins.certificate.CertificateProvider; | ||
import com.amazon.dataprepper.plugins.certificate.file.FileCertificateProvider; | ||
import com.amazon.dataprepper.plugins.source.loghttp.HTTPSourceConfig; | ||
import org.slf4j.Logger; | ||
import org.slf4j.LoggerFactory; | ||
|
||
public class CertificateProviderFactory { | ||
private static final Logger LOG = LoggerFactory.getLogger(CertificateProviderFactory.class); | ||
|
||
final HTTPSourceConfig httpSourceConfig; | ||
public CertificateProviderFactory(final HTTPSourceConfig httpSourceConfig) { | ||
this.httpSourceConfig = httpSourceConfig; | ||
} | ||
|
||
public CertificateProvider getCertificateProvider() { | ||
// TODO: support more certificate providers | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Make an issue for this, please. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
LOG.info("Using local file system to get certificate and private key for SSL/TLS."); | ||
return new FileCertificateProvider(httpSourceConfig.getSslCertificateFile(), httpSourceConfig.getSslKeyFile()); | ||
} | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
-----BEGIN CERTIFICATE----- | ||
MIICHTCCAYYCCQD4hqYeYDQZADANBgkqhkiG9w0BAQUFADBSMQswCQYDVQQGEwJV | ||
UzELMAkGA1UECAwCVFgxDzANBgNVBAcMBkF1c3RpbjEPMA0GA1UECgwGQW1hem9u | ||
MRQwEgYDVQQLDAtEYXRhcHJlcHBlcjAgFw0yMTA2MjUxOTIzMTBaGA8yMTIxMDYw | ||
MTE5MjMxMFowUjELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAlRYMQ8wDQYDVQQHDAZB | ||
dXN0aW4xDzANBgNVBAoMBkFtYXpvbjEUMBIGA1UECwwLRGF0YXByZXBwZXIwgZ8w | ||
DQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAKrb3YhdKbQ5PtLHall10iLZC9ZdDVrq | ||
HOvqVSM8NHlL8f82gJ3l0n9k7hYc5eKisutaS9eDTmJ+Dnn8xn/qPSKTIq9Wh+OZ | ||
O+e9YEEpI/G4F9KpGULgMyRg9sJK0GlZdEt9o5GJNJIJUkptJU5eiLuE0IV+jyJo | ||
Nvm8OE6EJPqxAgMBAAEwDQYJKoZIhvcNAQEFBQADgYEAjgnX5n/Tt7eo9uakIGAb | ||
uBhvYdR8JqKXqF9rjFJ/MIK7FdQSF/gCdjnvBhzLlZFK/Nb6MGKoSKm5Lcr75LgC | ||
FyhIwp3WlqQksiMFnOypYVY71vqDgj6UKdMaOBgthsYhngj8lC+wsVzWqQvkJ2Qg | ||
/GAIzJwiZfXiaevQHRk79qI= | ||
-----END CERTIFICATE----- |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
-----BEGIN RSA PRIVATE KEY----- | ||
MIICXAIBAAKBgQCq292IXSm0OT7Sx2pZddIi2QvWXQ1a6hzr6lUjPDR5S/H/NoCd | ||
5dJ/ZO4WHOXiorLrWkvXg05ifg55/MZ/6j0ikyKvVofjmTvnvWBBKSPxuBfSqRlC | ||
4DMkYPbCStBpWXRLfaORiTSSCVJKbSVOXoi7hNCFfo8iaDb5vDhOhCT6sQIDAQAB | ||
AoGANrrhFqpJDpr7vcb1ER0Fp/YArbT27zVo+EUC6puBb41dQlQyFOImcHpjLaAq | ||
H1PgnjU5cBp2hGQ+vOK0rwrYc/HNl6vfh6N3NbDptMiuoBafRJA9JzYourAM09BU | ||
zmXyr61Yn3KHzx1PRwWe37icX93oXP3P0qHb3dI1ZF4jG0ECQQDU5N/a7ogoz2zn | ||
ZssD6FvUOUQDsdBWdXmhUvg+YdZrV44e4xk+FVzwEONoRktEYKz9MFXlsgNHr445 | ||
KRguHWcJAkEAzXQkwOkN8WID1wrwoobUIMbZSGAZzofwkKXgTTnllnT1qOQXuRbS | ||
aCMejFEymBBef4aXP6N4+va2FKW/MF34aQJAO2oMl1sOoOUSrZngepy0VAwPUUCk | ||
thxe74jqQu6nGpn6zd/vQYZQw6bS8Fz90H1yic6dilcd1znFZWp0lxoZkQJBALeI | ||
xoBycRsuFQIYasi1q3AwUtBd0Q/3zkZZeBtk2hzjFMUwJaUZpxKSNOrialD/ZnuD | ||
jz+xWBTRKe0d98JMX+kCQCmsJEj/HYQAC1GamZ7JQWogRSRF2KTgTWRaDXDxy0d4 | ||
yUQgwHB+HZLFcbi1JEK6eIixCsX8iifrrkteh+1npJ0= | ||
-----END RSA PRIVATE KEY----- |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nothing you have to fix now, but going forward, please use single quotes for Gradle strings. Double quotes are GStrings which are used for string interpolation.