This article demonstrates how to use the HTML Parser to parse HTML files.
The HTML parser takes any fragment of HTML, uri or a stream and trys to parse it into a DOM. The parser is based on the HTML Living Standard Once a document/fragment has been parsed, a set of extension methods over the HTML DOM elements allow you to extract information from a web page independently of the actual HTML Type provider.
open FSharp.Data
The following example uses Google to search for FSharp.Data and then parses the first set of
search results from the page, extracting the URL and Title of the link.
We use the HtmlDocument type.
To achieve this we must first parse the webpage into our DOM. We can do this using the HtmlDocument.Load method. This method will take a URL and make a synchronous web call to extract the data from the page. Note: an asynchronous variant HtmlDocument.AsyncLoad is also available
let results = HtmlDocument.Load("http://www.google.co.uk/search?q=FSharp.Data")
val results: HtmlDocument =
<!DOCTYPE html>
<html lang="en">
<head>
<title>Google Search</title><style>body{background-color:#fff}</style><script nonce="XpNDRgpT12Y_K_eHpCQWww">window.google = window.google || {};window.google.c = window.google.c || {cap:0};</script>
</head>
<body>
<noscript>
<style>table,div,span,p{display:none}</style><meta content="0;url=/httpservice/retry/enablejs?sei=6jXVabqrHMLEp84Pi9zAiAM" http-equiv="refresh" />
<div style="display:block">
Please click <a href="/ht...Now that we have a loaded HTML document we can begin to extract data from it.
Firstly, we want to extract all of the anchor tags a out of the document, then
inspect the links to see if it has a href attribute, using HtmlDocumentExtensions.Descendants. If it does, extract the value,
which in this case is the url that the search result is pointing to, and additionally the
InnerText of the anchor tag to provide the name of the web page for the search result
we are looking at.
let links =
results.Descendants [ "a" ]
|> Seq.choose (fun x -> x.TryGetAttribute("href") |> Option.map (fun a -> x.InnerText(), a.Value()))
|> Seq.truncate 10
|> Seq.toList
val links: (string * string) list =
[("here", "/httpservice/retry/enablejs?sei=6jXVabqrHMLEp84Pi9zAiAM");
("click here",
"/search?q=FSharp.Data&sca_esv=38623c8ca84aacad&ie=UTF-8&emsg="+[34 chars]);
("feedback", "https://support.google.com/websearch")]Now that we have extracted our search results you will notice that there are lots of
other links to various Google services and cached/similar results. Ideally, we would
like to filter these results as we are probably not interested in them.
At this point we simply have a sequence of Tuples, so F# makes this trivial using Seq.filter
and Seq.map.
let searchResults =
links
|> List.filter (fun (name, url) -> name <> "Cached" && name <> "Similar" && url.StartsWith("/url?"))
|> List.map (fun (name, url) -> name, url.Substring(0, url.IndexOf("&sa=")).Replace("/url?q=", ""))
val searchResults: (string * string) list = []Multiple items
namespace FSharp
--------------------
namespace Microsoft.FSharp
Multiple items
namespace FSharp.Data
--------------------
namespace Microsoft.FSharp.Data
val results: HtmlDocument
Multiple items
module HtmlDocument
from FSharp.Data
--------------------
type HtmlDocument =
private | HtmlDocument of docType: string * elements: HtmlNode list
override ToString: unit -> string
static member New: docType: string * children: HtmlNode seq -> HtmlDocument + 1 overload
static member HtmlDocument.Load: stream: System.IO.Stream -> HtmlDocument
static member HtmlDocument.Load: reader: System.IO.TextReader -> HtmlDocument
static member HtmlDocument.Load: uri: string * ?encoding: System.Text.Encoding -> HtmlDocument
val links: (string * string) list
static member HtmlDocumentExtensions.Descendants: doc: HtmlDocument -> HtmlNode seq
static member HtmlDocumentExtensions.Descendants: doc: HtmlDocument * predicate: (HtmlNode -> bool) -> HtmlNode seq
static member HtmlDocumentExtensions.Descendants: doc: HtmlDocument * names: string seq -> HtmlNode seq
static member HtmlDocumentExtensions.Descendants: doc: HtmlDocument * name: string -> HtmlNode seq
static member HtmlDocumentExtensions.Descendants: doc: HtmlDocument * predicate: (HtmlNode -> bool) * recurseOnMatch: bool -> HtmlNode seq
static member HtmlDocumentExtensions.Descendants: doc: HtmlDocument * names: string seq * recurseOnMatch: bool -> HtmlNode seq
static member HtmlDocumentExtensions.Descendants: doc: HtmlDocument * name: string * recurseOnMatch: bool -> HtmlNode seq
module Seq from Microsoft.FSharp.Collections
val choose: chooser: ('T -> 'U option) -> source: 'T seq -> 'U seq
val x: HtmlNode
static member HtmlNodeExtensions.TryGetAttribute: n: HtmlNode * name: string -> HtmlAttribute option
module Option from Microsoft.FSharp.Core
val map: mapping: ('T -> 'U) -> option: 'T option -> 'U option
val a: HtmlAttribute
static member HtmlNodeExtensions.InnerText: n: HtmlNode -> string
static member HtmlAttributeExtensions.Value: attr: HtmlAttribute -> string
val truncate: count: int -> source: 'T seq -> 'T seq
val toList: source: 'T seq -> 'T list
val searchResults: (string * string) list
Multiple items
module List
from Microsoft.FSharp.Collections
--------------------
type List<'T> =
| op_Nil
| op_ColonColon of Head: 'T * Tail: 'T list
interface IReadOnlyList<'T>
interface IReadOnlyCollection<'T>
interface IEnumerable
interface IEnumerable<'T>
member GetReverseIndex: rank: int * offset: int -> int
member GetSlice: startIndex: int option * endIndex: int option -> 'T list
static member Cons: head: 'T * tail: 'T list -> 'T list
member Head: 'T
member IsEmpty: bool
member Item: index: int -> 'T with get
...
val filter: predicate: ('T -> bool) -> list: 'T list -> 'T list
val name: string
val url: string
System.String.StartsWith(value: string) : bool
System.String.StartsWith(value: char) : bool
System.String.StartsWith(value: string, comparisonType: System.StringComparison) : bool
System.String.StartsWith(value: string, ignoreCase: bool, culture: System.Globalization.CultureInfo) : bool
val map: mapping: ('T -> 'U) -> list: 'T list -> 'U list
System.String.Substring(startIndex: int) : string
System.String.Substring(startIndex: int, length: int) : string
System.String.IndexOf(value: string) : int
System.String.IndexOf(value: char) : int
System.String.IndexOf(value: string, comparisonType: System.StringComparison) : int
System.String.IndexOf(value: string, startIndex: int) : int
System.String.IndexOf(value: char, comparisonType: System.StringComparison) : int
System.String.IndexOf(value: char, startIndex: int) : int
System.String.IndexOf(value: string, startIndex: int, comparisonType: System.StringComparison) : int
System.String.IndexOf(value: string, startIndex: int, count: int) : int
System.String.IndexOf(value: char, startIndex: int, count: int) : int
System.String.IndexOf(value: string, startIndex: int, count: int, comparisonType: System.StringComparison) : int