GitHub - nkw/FSharp.Data: F# Data: Library for Data Access

graph TD
    subgraph "Client Application (e.g., Web API)"
        API["API Controller"]
    end

    subgraph "Orleans Cluster"
        direction TB
        Ingestion["Ingestion Grains<br>[StatelessWorker]<br>No future Dates"]
        Manager["CacheManagerGrain<br>(Singleton Timer)"]

        subgraph "Rotating Cache Slots (4 Grains)"
            S0["Slot 0"]
            S1["Slot 1"]
            S2["Slot 2"]
            S3["Slot 3"]
        end
    end

    subgraph "External Systems"
        Kafka["Kafka Topic"]
        DB["Persistent Store (DB)"]
    end

    %% FLOW 1: Data Ingestion (Kafka-Only)
    Kafka -- "Data Stream" --> Ingestion
    Ingestion -- "Appends data to correct slot<br>based on DayNumber" --> S0
    Ingestion --> S1
    Ingestion --> S2
    Ingestion --> S3

    %% FLOW 2: Client Request for Recent Data (within 4 days)
    API -- "Gets data from correct slot<br>based on DayNumber" --> S0
    API --> S1
    API --> S2
    API --> S3

    %% FLOW 3: Client Request for Old Data (> 4 days)
    API -- "Bypasses cache, reads from DB" --> DB

    %% FLOW 4: Cache Management (Daily Purge)
    Manager -- "Daily: Purges oldest slot & prepares it for new day's data" --> S0 & S1 & S2 & S3
Loading

The F# Data library (FSharp.Data.dll) implements everything you need to access data in your F# applications and scripts. It implements F# type providers for working with structured file formats (CSV, JSON and XML) and for accessing the WorldBank and Freebase services. It also includes helpers for parsing JSON files and for sending HTTP requests.

We're open to contributions from anyone. If you want to help out but don't know where to start, you can take one of the Up-For-Grabs issues, or help to improve the documentation.

You can see the version history here.

Building

  • Simply build FSharp.Data.sln in Visual Studio, Mono Develop, or Xamarin Studio. You can also use the FAKE script:

Supported platforms

  • VS2012 compiling to FSharp.Core 4.3.0.0
  • VS2012 compiling to FSharp.Core 2.3.5.0 (PCL profile 47)
  • Mono F# 3.0 compiling to FSharp.Core 4.3.0.0
  • Mono F# 3.0 compiling to FSharp.Core 2.3.5.0 (PCL profile 47)
  • VS2013 compiling to FSharp.Core 4.3.0.0
  • VS2013 compiling to FSharp.Core 4.3.1.0
  • VS2013 compiling to FSharp.Core 2.3.5.0 (PCL profile 47)
  • VS2013 compiling to FSharp.Core 2.3.6.0 (PCL profile 47)
  • Mono F# 3.1 compiling to FSharp.Core 4.3.0.0
  • Mono F# 3.1 compiling to FSharp.Core 4.3.1.0
  • Mono F# 3.1 compiling to FSharp.Core 2.3.5.0 (PCL profile 47)
  • Mono F# 3.1 compiling to FSharp.Core 2.3.6.0 (PCL profile 47)

Documentation

This library is that it comes with comprehensive documentation. The documentation is automatically generated from *.fsx files in the content folder and from the comments in the code. If you find a typo, please submit a pull request!

  • F# Data Library home page with more information about the library, contributions, etc.
  • The samples from the documentation are included as part of FSharp.Data.Tests.sln, make sure you build the solution before trying out the samples to ensure that all needed packages are installed.

Support and community

Library license

The library is available under Apache 2.0. For more information see the License file in the GitHub repository.