Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[large-tiles] Large tiles aggregation flow #2617

Merged
merged 20 commits into from
Sep 14, 2020
Merged

[large-tiles] Large tiles aggregation flow #2617

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
c4dc67b
Merge branch 'master' into large-tiles-aggregation
linasm Jul 17, 2020
aad2375
Merge branch 'master' into large-tiles-aggregation
linasm Aug 5, 2020
f9c465d
[dbnode] Add OrderedByIndex option for DataFileSetReader.Open (#2465)
linasm Aug 11, 2020
81ce0ed
[dbnode] Cross-block series reader (#2481)
linasm Aug 11, 2020
0dbbff1
[dbnode] AggregateTiles RPC - minimal E2E flow (#2466)
linasm Aug 19, 2020
9916461
Merge branch 'master' into large-tiles-aggregation
linasm Aug 20, 2020
cbefee1
Merge branch 'master' into large-tiles-aggregation
linasm Sep 8, 2020
f5e8caf
Merge branch 'master' into large-tiles-aggregation
linasm Sep 9, 2020
0bc9d32
Fix TestReadAggregateWrite
linasm Sep 9, 2020
7acdeda
typo
arnikola Sep 9, 2020
bb87023
Convert spaces to tabs in rpc.thrift
linasm Sep 10, 2020
6fe7338
Change capitalization of source/target namespace
linasm Sep 10, 2020
dd26813
Fix copyright year in large_tiles_test.go
linasm Sep 10, 2020
8d4e70c
Remove a noop
linasm Sep 10, 2020
ffcc058
PR feedback
linasm Sep 10, 2020
491c93d
PR feedback
linasm Sep 10, 2020
718f086
More PR feedback
linasm Sep 10, 2020
7f402b0
Check bounds on dataMmap access
linasm Sep 14, 2020
b1fedbb
Renamed metrics.writeAggregated to metrics.aggregateTiles
linasm Sep 14, 2020
981d92e
Merge branch 'master' into large-tiles-aggregation
linasm Sep 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/dbnode/generated/mocks/generate.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@

// mockgen rules for generating mocks for exported interfaces (reflection mode)

//go:generate sh -c "mockgen -package=fs $PACKAGE/src/dbnode/persist/fs DataFileSetWriter,DataFileSetReader,DataFileSetSeeker,IndexFileSetWriter,IndexFileSetReader,IndexSegmentFileSetWriter,IndexSegmentFileSet,IndexSegmentFile,SnapshotMetadataFileWriter,DataFileSetSeekerManager,ConcurrentDataFileSetSeeker,MergeWith | genclean -pkg $PACKAGE/src/dbnode/persist/fs -out $GOPATH/src/$PACKAGE/src/dbnode/persist/fs/fs_mock.go"
//go:generate sh -c "mockgen -package=fs $PACKAGE/src/dbnode/persist/fs DataFileSetWriter,DataFileSetReader,DataFileSetSeeker,IndexFileSetWriter,IndexFileSetReader,IndexSegmentFileSetWriter,IndexSegmentFileSet,IndexSegmentFile,SnapshotMetadataFileWriter,DataFileSetSeekerManager,ConcurrentDataFileSetSeeker,MergeWith,CrossBlockReader | genclean -pkg $PACKAGE/src/dbnode/persist/fs -out $GOPATH/src/$PACKAGE/src/dbnode/persist/fs/fs_mock.go"
//go:generate sh -c "mockgen -package=xio $PACKAGE/src/dbnode/x/xio SegmentReader,SegmentReaderPool | genclean -pkg $PACKAGE/src/dbnode/x/xio -out $GOPATH/src/$PACKAGE/src/dbnode/x/xio/io_mock.go"
//go:generate sh -c "mockgen -package=digest -destination=$GOPATH/src/$PACKAGE/src/dbnode/digest/digest_mock.go $PACKAGE/src/dbnode/digest ReaderWithDigest"
//go:generate sh -c "mockgen -package=series $PACKAGE/src/dbnode/storage/series DatabaseSeries,QueryableBlockRetriever | genclean -pkg $PACKAGE/src/dbnode/storage/series -out $GOPATH/src/$PACKAGE/src/dbnode/storage/series/series_mock.go"
Expand Down
16 changes: 16 additions & 0 deletions src/dbnode/generated/thrift/rpc.thrift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@ service Node {
void repair() throws (1: Error err)
TruncateResult truncate(1: TruncateRequest req) throws (1: Error err)

AggregateTilesResult aggregateTiles(1: AggregateTilesRequest req) throws (1: Error err)

// Management endpoints
NodeHealthResult health() throws (1: Error err)
// NB: bootstrapped is for use with cluster management tools like k8s.
Expand Down Expand Up @@ -490,6 +492,20 @@ struct Query {
7: optional FieldQuery field
}

struct AggregateTilesRequest {
1: required string sourceNameSpace
2: required string targetNameSpace
3: required i64 rangeStart
4: required i64 rangeEnd
5: required string step
6: bool removeResets
7: optional TimeType rangeType = TimeType.UNIX_SECONDS
}

struct AggregateTilesResult {
1: required i64 processedBlockCount
}

struct DebugProfileStartRequest {
1: required string name
2: required string filePathTemplate
Expand Down
Loading