Bucket Notifications #8337
Merged
Bucket Notifications #8337
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alphaprinz
force-pushed
the
notifications
branch
2 times, most recently
from
180b6fc to
e3888e6
Compare
alphaprinz
force-pushed
the
notifications
branch
4 times, most recently
from
7c2633e to
e48b9c5
Compare
alphaprinz
force-pushed
the
notifications
branch
from
e48b9c5 to
c84729b
Compare
alphaprinz
force-pushed
the
notifications
branch
5 times, most recently
from
d09a342 to
47edd81
Compare
alphaprinz
force-pushed
the
notifications
branch
3 times, most recently
from
a090200 to
627dd18
Compare
dannyzaken
reviewed
Oct 10, 2024
| @@ -76,6 +79,11 @@ async function put_object(req, res) { | |||
| }; | |||
| } | |||
| res.setHeader('ETag', `"${reply.etag}"`); | |||
|
|
|||
| if (reply.seq) { | |||
| res.seq = reply.seq; | |||
There was a problem hiding this comment.
Seems like seq is only returned in namespace_nb flows. This will not work for other namespaces
There was a problem hiding this comment.
Right.
As we've discussed we can add a sequence to standalone, at the cost of adding a synchronized step to all s3 ops, so it's preferred for now to see if this is actually needed.
There was a problem hiding this comment.
But what about other namespaces (S3, Azure, FS) that are not in standalone?
alphaprinz
force-pushed
the
notifications
branch
3 times, most recently
from
e81d8dd to
cdcf436
Compare
alphaprinz
force-pushed
the
notifications
branch
3 times, most recently
from
fe4db12 to
cb3a0c5
Compare
alphaprinz
force-pushed
the
notifications
branch
3 times, most recently
from
af9d0f6 to
eb831a2
Compare
Signed-off-by: Amit Prinz Setter <[email protected]>
Signed-off-by: Amit Prinz Setter <[email protected]>
Signed-off-by: Amit Prinz Setter <[email protected]>
Signed-off-by: Amit Prinz Setter <[email protected]>
Signed-off-by: Amit Prinz Setter <[email protected]>
…amespace for easier parsing) Signed-off-by: Amit Prinz Setter <[email protected]>
dannyzaken
approved these changes
Oct 30, 2024
1. Introduce OP_TO_EVENT, remove req.s3event 2. sequencer fallback to timestamp for non-noobaa nss Signed-off-by: Amit Prinz Setter <[email protected]>
alphaprinz
force-pushed
the
notifications
branch
from
eb831a2 to
6c40606
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Explain the changes
Add S3 notification support.
AWS info https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html
Limitations:
-No LifeCycle events.
Issues: Fixed #xxx / Gap #xxx
Testing Instructions:
-Run local Kafka with
docker run -p 9092:9092 apache/kafka-native:3.8.0
Connect cli consumer to Kafka:
./kafka/kafka_2.13-3.8.0/bin/kafka-console-consumer.sh --topic --from-beginning --bootstrap-server localhost:9092
Alternatively, for nsfs, you can use nsfs_manage to add "notifications" field to bucket.
Example of an nsfs bucket with notifications:
aprinzse:noobaa-core$ cat bucket_notif_event.json
{
"name": "notif-both",
"owner": "amit1",
"path": "/home/aprinzse/work/standalone/storage_root/notif_both",
"notifications": [
{"Id": "notif_event_kafka", "Events": ["s3:ObjectCreated:"], "Connect": "/home/aprinzse/work/standalone/connect.kv"},
{"Id": "notif_event_http", "Events": ["s3:ObjectCreated:"], "Connect": "/home/aprinzse/work/standalone/connect_http.kv"}
]
}
Where connect.kv for Kafka looks like-
aprinzse:noobaa-core$ cat /home/aprinzse/work/standalone/connect.kv
metadata.broker.list=localhost:9092
notification_protocol=kafka
topic=
name=kafka_notif
And for http:
aprinzse:noobaa-core$ cat /home/aprinzse/work/standalone/connect_http.kv
http_object={"host": "localhost", "port": 9999, "timeout": 100}
notification_protocol=http
name=http_notif
-Run NSFS server with NOTIFICATION_LOG_DIR env, eg
NOTIFICATION_LOG_DIR=/home/aprinzse/work/standalone/notifications_log node src/cmd/nsfs
-Kicking-off notification file processor to send notification with "notification" operation (also need the NOTIFICATION_LOG_DIR env param):
NOTIFICATION_LOG_DIR=/home/aprinzse/work/standalone/notifications_log node src/cmd/manage_nsfs notification