[BEAM-14267] Update watchForNewFiles to allow watching updated files by Abacn · Pull Request #17305 · apache/beam

@Abacn

* Added overload versions of continuously for FileIO.match and .matchAll
  accepting a third matchUpdatedFiles parameter.

* Watching for both new files and timestamp change of existing files
  when matchUpdatedFiles is set as true

@Abacn

@Abacn

* Fix timing

* Update FileIO.populateDisplayData

johnjcasey

@Abacn

* Use a helper function for watch transfiorm

* Throw RuntimeException if extracting file timestamp fails

* Added GcsMatchIT testGcsMatchContinuously

@Abacn

johnjcasey

@Abacn

@Abacn

johnjcasey

@Abacn

@Abacn

@Abacn

@Abacn

@Abacn Abacn mentioned this pull request

Jun 14, 2022

@Abacn Abacn deleted the watchforfileupdate branch

June 14, 2022 16:58

bullet03 pushed a commit to akvelon/beam that referenced this pull request

Jun 20, 2022
…pache#17305)

* [BEAM-14267] Update watchForNewFiles to allow watching updated files

* Added overload versions of continuously for FileIO.match and .matchAll
  accepting a third matchUpdatedFiles parameter.

* Watching for both new files and timestamp change of existing files
  when matchUpdatedFiles is set as true

* run spotlessApply

* Fix Java Pre Commit

* Fix timing

* Update FileIO.populateDisplayData

* Address comments and add integration test

* Use a helper function for watch transfiorm

* Throw RuntimeException if extracting file timestamp fails

* Added GcsMatchIT testGcsMatchContinuously

* Move GcsMatchIT to correct path

* Add comment specifying the watch freq

* method naming; descriptive exception message

* Fix precommit checkstyleTest

* Fix test flacky in Dataflow

* Fix outdated comment

* Add dumb commit (a piece of comment for test) to trigger testing for PR