[BEAM-14267] Update watchForNewFiles to allow watching updated files by Abacn · Pull Request #17305 · apache/beam
* Added overload versions of continuously for FileIO.match and .matchAll accepting a third matchUpdatedFiles parameter. * Watching for both new files and timestamp change of existing files when matchUpdatedFiles is set as true
* Use a helper function for watch transfiorm * Throw RuntimeException if extracting file timestamp fails * Added GcsMatchIT testGcsMatchContinuously
Abacn
mentioned this pull request
Abacn
deleted the
watchforfileupdate
branch
bullet03 pushed a commit to akvelon/beam that referenced this pull request
Jun 20, 2022…pache#17305) * [BEAM-14267] Update watchForNewFiles to allow watching updated files * Added overload versions of continuously for FileIO.match and .matchAll accepting a third matchUpdatedFiles parameter. * Watching for both new files and timestamp change of existing files when matchUpdatedFiles is set as true * run spotlessApply * Fix Java Pre Commit * Fix timing * Update FileIO.populateDisplayData * Address comments and add integration test * Use a helper function for watch transfiorm * Throw RuntimeException if extracting file timestamp fails * Added GcsMatchIT testGcsMatchContinuously * Move GcsMatchIT to correct path * Add comment specifying the watch freq * method naming; descriptive exception message * Fix precommit checkstyleTest * Fix test flacky in Dataflow * Fix outdated comment * Add dumb commit (a piece of comment for test) to trigger testing for PR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters