` (table row), `` (table heading), and ` ` (table data).
Here's a simple HTML table with two columns and three rows:
```{r}
html <- minimal_html("
x
y
1.5
2.7
4.9
1.3
7.2
8.1
")
```
Because tables are a common way to store data, rvest includes the handy `html_table()` which converts a table into a data frame:
```{r}
html |>
html_node("table") |>
html_table()
```
## Element vs elements
When using rvest, your eventual goal is usually to build up a data frame, and you want each row to correspond some repeated unit on the HTML page.
In this case, you should generally start by using `html_elements()` to select the elements that contain each observation then use `html_element()` to extract the variables from each observation.
This guarantees that you'll get the same number of values for each variable because `html_element()` always returns the same number of outputs as inputs.
To illustrate this problem take a look at this simple example I constructed using a few entries from `dplyr::starwars`:
```{r}
html <- minimal_html("
C-3PO is a droid that weighs 167 kg R2-D2 is a droid that weighs 96 kg Yoda weighs 66 kg R4-P17 is a droid
")
```
If you try to extract name, species, and weight directly, you end up with one vector of length four and two vectors of length three, and no way to align them:
```{r}
html |> html_elements("b") |> html_text2()
html |> html_elements("i") |> html_text2()
html |> html_elements(".weight") |> html_text2()
```
Instead, use `html_elements()` to find a element that corresponds to each character, then use `html_element()` to extract each variable for all observations:
```{r}
characters <- html |> html_elements("li")
characters |> html_element("b") |> html_text2()
characters |> html_element("i") |> html_text2()
characters |> html_element(".weight") |> html_text2()
```
`html_element()` automatically fills in `NA` when no elements match, keeping all of the variables aligned and making it easy to create a data frame:
```{r}
data.frame(
name = characters |> html_element("b") |> html_text2(),
species = characters |> html_element("i") |> html_text2(),
weight = characters |> html_element(".weight") |> html_text2()
)
```
rvest/vignettes/starwars.Rmd 0000644 0001762 0000144 00000001701 15051552265 015744 0 ustar ligges users ---
title: "Star Wars films (static HTML)"
output: rmarkdown::html_vignette
vignette: >
%\VignetteIndexEntry{Star Wars films (static HTML)}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
---
This vignette contains some data about the Star Wars films for use in rvest examples and vignettes.
```{r, echo = FALSE, results = "asis"}
library(rvest)
crawl_html <- function(x) {
out <- gsub("\r", "", x)
out <- gsub("\n\n", "", out)
out <- gsub("\n", " ", out)
paste0("
", out, "
")
}
film_desc <- function(x) {
glue::glue_data(x, "
{title}
Released: {release_date}
Director: {director}
{crawl_html(opening_crawl)}
")
}
films <- repurrrsive::sw_films
films <- films[order(sapply(films, "[[", "episode_id"))]
descs <- vapply(films, film_desc, character(1))
writeLines(descs)
```
rvest/NAMESPACE 0000644 0001762 0000144 00000004116 14557166073 012654 0 ustar ligges users # Generated by roxygen2: do not edit by hand
S3method(cookies,rvest_session)
S3method(format,rvest_field)
S3method(headers,rvest_session)
S3method(html_element,LiveHTML)
S3method(html_element,default)
S3method(html_element,rvest_session)
S3method(html_elements,LiveHTML)
S3method(html_elements,default)
S3method(html_elements,rvest_session)
S3method(html_form,rvest_session)
S3method(html_form,xml_document)
S3method(html_form,xml_node)
S3method(html_form,xml_nodeset)
S3method(html_table,LiveHTML)
S3method(html_table,rvest_session)
S3method(html_table,xml_document)
S3method(html_table,xml_node)
S3method(html_table,xml_nodeset)
S3method(html_text2,xml_document)
S3method(html_text2,xml_missing)
S3method(html_text2,xml_node)
S3method(html_text2,xml_nodeset)
S3method(print,rvest_field)
S3method(print,rvest_form)
S3method(print,rvest_session)
S3method(read_html,rvest_session)
S3method(status_code,rvest_session)
export("%>%")
export(LiveHTML)
export(back)
export(follow_link)
export(forward)
export(google_form)
export(guess_encoding)
export(html_attr)
export(html_attrs)
export(html_children)
export(html_element)
export(html_elements)
export(html_encoding_guess)
export(html_form)
export(html_form_set)
export(html_form_submit)
export(html_name)
export(html_node)
export(html_nodes)
export(html_session)
export(html_table)
export(html_text)
export(html_text2)
export(is.session)
export(jump_to)
export(minimal_html)
export(read_html)
export(read_html_live)
export(repair_encoding)
export(session)
export(session_back)
export(session_follow_link)
export(session_forward)
export(session_history)
export(session_jump_to)
export(session_submit)
export(set_values)
export(submit_form)
export(url_absolute)
export(xml_node)
export(xml_nodes)
export(xml_tag)
import(rlang)
importFrom(glue,glue)
importFrom(httr,cookies)
importFrom(httr,headers)
importFrom(httr,status_code)
importFrom(lifecycle,deprecated)
importFrom(magrittr,"%>%")
importFrom(xml2,read_html)
importFrom(xml2,url_absolute)
importFrom(xml2,xml_attr)
importFrom(xml2,xml_attrs)
importFrom(xml2,xml_children)
importFrom(xml2,xml_name)
importFrom(xml2,xml_text)
rvest/LICENSE 0000644 0001762 0000144 00000000053 14554031036 012421 0 ustar ligges users YEAR: 2023
COPYRIGHT HOLDER: rvest authors
rvest/NEWS.md 0000644 0001762 0000144 00000020775 15054321657 012536 0 ustar ligges users # rvest 1.0.5
* Fixes `LiveHTML` objects returning 'could not find node with given id' errors if a page navigation occurred after a `$click()` event (#405).
* New example vignette displays the same starwars data but rendered dynamically using JS, so you need to use `read_html_live()` to get the data.
# rvest 1.0.4
* New `read_html_live()` reads HTML into a real, live, HTML browser, meaning
that you can scrape HTML generated by javascript. It returns a `LiveHTML`
object which you can also use to simulate user interactions with the page,
like clicking, typing, and scrolling (#245).
* `html_table()` discards rows without cells (@epiben, #360).
# rvest 1.0.3
* Re-document to fix HTML issues in `.Rd`.
# rvest 1.0.2
* Fixes for CRAN
* `html_table()` converts empty tables to empty tibbles (@epiben, #327).
# rvest 1.0.1
* `html_table()` correctly handles tables with cells that contain blank values
for `rowspan` and/or `colspan`, so that e.g. ` ` is parsed as
` ` (@epiben, #323).
* Fix broken example
# rvest 1.0.0
## New features
* New `html_text2()` provides a more natural rendering of HTML nodes into text,
converting `,
so we can be more generous with screenshots since they're no longer bundled
with every install of the package. Together with the rewrite of the other
vignette, this means that rvest is now ~90 Kb instead of ~1.1 Mb.
* All uses of IMDB have been eliminated since the site explicitly prohibits
scraping (#195).
* `session_submit()` errors if `form` doesn't have a `url` (#288).
* New `session_forward()` function to complement `session_back()`.
It now allows you to pick the submission button by position (#156).
The `...` argument is deprecated; please use `config` instead.
* `html_form_set()` can now accept character vectors allowing you to select
multiple checkboxes in a set or select multiple values from a multi-``
(#127, with help from @juba). It also uses dynamic dots so that you can use
`!!!` if you have a list of values (#189).
# rvest 0.3.6
* Remove failing example
# rvest 0.3.5
* Use web archive to fix broken example.
# rvest 0.3.4
* Remove unneeded `read_xml.response()` method (#242).
# rvest 0.3.3
* Fix `R CMD check` failure
* `submit_request()` now checks for empty form-field-types to select the
correct submit fields (@rentrop, #159)
# rvest 0.3.2
* Fixes to `follow_link()` and `back()` to correctly manage session history.
* If you're using xml2 1.0.0, `html_node()` will now return a "missing node".
* Parse rowspans and colspans effectively by filling using repetition from
left to right (for colspan) and top to bottom (rowspan) (#111)
* Updated a few examples and demos where the website structure has
changed.
* Made compatible with both xml2 0.1.2 and 1.0.0.
# rvest 0.3.1
* Fix invalid link for SSA example.
* Parse `` that don't have value attribute (#85).
* Remove all remaining uses of `html()` in favor of `read_html()`
(@jimhester, #113).
# rvest 0.3.0
* rvest has been rewritten to take advantage of the new xml2 package. xml2
provides a fresh binding to libxml2, avoiding many of the work-arounds
previously needed for the XML package. Now rvest depends on the xml2
package, so all the xml functions are available, and rvest adds a thin
wrapper for html.
* A number of functions have change names. The old versions still work,
but are deprecated and will be removed in rvest 0.4.0.
* `html_tag()` -> `html_name()`
* `html()` -> `read_html()`
* `html_node()` now throws an error if there are no matches, and a warning
if there's more than one match. I think this should make it more likely to
fail clearly when the structure of the page changes.
* `xml_structure()` has been moved to xml2. New `html_structure()` (also in
xml2) highlights id and class attributes (#78).
* `submit_form()` now works with forms that use GET (#66).
* `submit_request()` (and hence `submit_form()`) is now case-insensitive,
and so will find `Bad encoding
migr cause clbre dj vu.
rvest/inst/doc/ 0000755 0001762 0000144 00000000000 15054321705 013141 5 ustar ligges users rvest/inst/doc/starwars-dynamic.Rmd 0000644 0001762 0000144 00000014177 15051552265 017113 0 ustar ligges users ---
title: "Star Wars films (dynamic HTML)"
output: rmarkdown::html_vignette
vignette: >
%\VignetteIndexEntry{Star Wars films (dynamic HTML)}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
---
```{r, eval = FALSE, echo = FALSE}
library(magrittr)
crawl_html <- function(x) {
x |>
gsub("\r", "", .) |>
gsub("\n\n", "", .) |>
gsub("\n", " ", .) |>
paste0("
", ., "
")
}
fields <- c("episode_id", "title", "release_date", "director", "opening_crawl")
repurrrsive::sw_films |>
lapply(\(film) film[fields]) |>
lapply(function(film) {
film$opening_crawl <- crawl_html(film$opening_crawl)
film
}) |>
jsonlite::write_json("vignettes/starwars.json", pretty = TRUE, auto_unbox = TRUE)
```
This vignette contains some dynamic generated data about the Star Wars films for use in rvest examples and vignettes.
```{=html}
```
::: {#moviesInfo}
:::
rvest/inst/doc/rvest.Rmd 0000644 0001762 0000144 00000026736 15051552265 014772 0 ustar ligges users ---
title: "Web scraping 101"
output: rmarkdown::html_vignette
vignette: >
%\VignetteIndexEntry{Web scraping 101}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
---
```{r, echo=FALSE}
knitr::opts_chunk$set(comment = "#>", collapse = TRUE)
```
This vignette introduces you to the basics of web scraping with rvest.
You'll first learn the basics of HTML and how to use CSS selectors to refer to specific elements, then you'll learn how to use rvest functions to get data out of HTML and into R.
```{r}
library(rvest)
```
## HTML basics
HTML stands for "HyperText Markup Language" and looks like this:
``` {.html}
Page title
A heading
Some text & some bold text.
`), optional **attributes** (`id='first'`), an end tag[^1] (like ` `), and **contents** (everything in between the start and end tag).
[^1]: A number of tags (including `` and `
)` don't require end tags, but I think it's best to include them because it makes seeing the structure of the HTML a little easier.
Since `<` and `>` are used for start and end tags, you can't write them directly.
Instead you have to use the HTML **escapes** `>` (greater than) and `<` (less than).
And since those escapes use `&`, if you want a literal ampersand you have to escape it as `&`.
There are a wide range of possible HTML escapes but you don't need to worry about them too much because rvest automatically handles them for you.
### Elements
All up, there are over 100 HTML elements.
Some of the most important are:
- Every HTML page must be in an `` element, and it must have two children: ``, which contains document metadata like the page title, and ``, which contains the content you see in the browser.
- Block tags like `` (heading 1), ` ` (paragraph), and `
` (ordered list) form the overall structure of the page.
- Inline tags like `` (bold), `` (italics), and `` (links) formats text inside block tags.
If you encounter a tag that you've never seen before, you can find out what it does with a little googling.
I recommend the [MDN Web Docs](https://developer.mozilla.org/en-US/docs/Web/HTML) which are produced by Mozilla, the company that makes the Firefox web browser.
### Contents
Most elements can have content in between their start and end tags.
This content can either be text or more elements.
For example, the following HTML contains paragraph of text, with one word in bold.
```{=html}
Hi! My name is Hadley.
```
The **children** of a node refers only to elements, so the `` element above has one child, the `` element.
The `` element has no children, but it does have contents (the text "name").
Some elements, like `
This is a paragraph
")
html
```
Regardless of how you get the HTML, you'll need some way to identify the elements that contain the data you care about.
rvest provides two options: CSS selectors and XPath expressions.
Here I'll focus on CSS selectors because they're simpler but still sufficiently powerful for most scraping tasks.
## CSS selectors
CSS is short for cascading style sheets, and is a tool for defining the visual styling of HTML documents.
CSS includes a miniature language for selecting elements on a page called **CSS selectors**.
CSS selectors define patterns for locating HTML elements, and are useful for scraping because they provide a concise way of describing which elements you want to extract.
CSS selectors can be quite complex, but fortunately you only need the simplest for rvest, because you can also write R code for more complicated situations.
The four most important selectors are:
- `p`: selects all `` elements.
- `.title`: selects all elements with `class` "title".
- `p.special`: selects all `
` elements with `class` "special".
- `#title`: selects the element with the `id` attribute that equals "title".
Id attributes must be unique within a document, so this will only ever select a single element.
If you want to learn more CSS selectors I recommend starting with the fun [CSS dinner](https://flukeout.github.io/) tutorial and then referring to the [MDN web docs](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors).
Lets try out the most important selectors with a simple example:
```{r}
html <- minimal_html("
This is a heading
This is a paragraph
This is an important paragraph
")
```
In rvest you can extract a single element with `html_element()` or all matching elements with `html_elements()`.
Both functions take a document[^3] and a css selector:
[^3]: Or another element, more on that shortly.
```{r}
html |> html_element("h1")
html |> html_elements("p")
html |> html_elements(".important")
html |> html_elements("#first")
```
Selectors can also be combined in various ways using **combinators**.
For example,The most important combinator is " ", the **descendant** combination, because `p a` selects all ` ` elements that are a child of a `` element.
If you don't know exactly what selector you need, I highly recommend using [SelectorGadget](https://rvest.tidyverse.org/articles/selectorgadget.html), which lets you automatically generate the selector you need by supplying positive and negative examples in the browser.
## Extracting data
Now that you've got the elements you care about, you'll need to get data out of them.
You'll usually get the data from either the text contents or an attribute.
But, sometimes (if you're lucky!), the data you need will be in an HTML table.
### Text
Use `html_text2()` to extract the plain text contents of an HTML element:
```{r}
html <- minimal_html("
apple & pear
banana
pineapple
")
html |>
html_elements("li") |>
html_text2()
```
Note that the escaped ampersand is automatically converted to `&`; you'll only ever see HTML escapes in the source HTML, not in the data returned by rvest.
You might wonder why I used `html_text2()`, since it seems to give the same result as `html_text()`:
```{r}
html |>
html_elements("li") |>
html_text()
```
The main difference is how the two functions handle white space.
In HTML, white space is largely ignored, and it's the structure of the elements that defines how text is laid out.
`html_text2()` does its best to follow the same rules, giving you something similar to what you'd see in the browser.
Take this example which contains a bunch of white space that HTML ignores.
```{r}
html <- minimal_html("
This is
a
paragraph.
This is another paragraph.
It has two sentences.
")
```
`html_text2()` gives you what you expect: two paragraphs of text separated by a blank line.
```{r}
html |>
html_element("body") |>
html_text2() |>
cat()
```
Whereas `html_text()` returns the garbled raw underlying text:
```{r}
html |>
html_element("body") |>
html_text() |>
cat()
```
### Attributes
Attributes are used to record the destination of links (the `href` attribute of ` ` elements) and the source of images (the `src` attribute of the `cats
`, `` (table row), `` (table heading), and ` ` (table data).
Here's a simple HTML table with two columns and three rows:
```{r}
html <- minimal_html("
x
y
1.5
2.7
4.9
1.3
7.2
8.1
")
```
Because tables are a common way to store data, rvest includes the handy `html_table()` which converts a table into a data frame:
```{r}
html |>
html_node("table") |>
html_table()
```
## Element vs elements
When using rvest, your eventual goal is usually to build up a data frame, and you want each row to correspond some repeated unit on the HTML page.
In this case, you should generally start by using `html_elements()` to select the elements that contain each observation then use `html_element()` to extract the variables from each observation.
This guarantees that you'll get the same number of values for each variable because `html_element()` always returns the same number of outputs as inputs.
To illustrate this problem take a look at this simple example I constructed using a few entries from `dplyr::starwars`:
```{r}
html <- minimal_html("
C-3PO is a droid that weighs 167 kg R2-D2 is a droid that weighs 96 kg Yoda weighs 66 kg R4-P17 is a droid
")
```
If you try to extract name, species, and weight directly, you end up with one vector of length four and two vectors of length three, and no way to align them:
```{r}
html |> html_elements("b") |> html_text2()
html |> html_elements("i") |> html_text2()
html |> html_elements(".weight") |> html_text2()
```
Instead, use `html_elements()` to find a element that corresponds to each character, then use `html_element()` to extract each variable for all observations:
```{r}
characters <- html |> html_elements("li")
characters |> html_element("b") |> html_text2()
characters |> html_element("i") |> html_text2()
characters |> html_element(".weight") |> html_text2()
```
`html_element()` automatically fills in `NA` when no elements match, keeping all of the variables aligned and making it easy to create a data frame:
```{r}
data.frame(
name = characters |> html_element("b") |> html_text2(),
species = characters |> html_element("i") |> html_text2(),
weight = characters |> html_element(".weight") |> html_text2()
)
```
rvest/inst/doc/starwars.R 0000644 0001762 0000144 00000001257 15054321704 015136 0 ustar ligges users ## ----echo = FALSE, results = "asis"-------------------------------------------
library(rvest)
crawl_html <- function(x) {
out <- gsub("\r", "", x)
out <- gsub("\n\n", "", out)
out <- gsub("\n", " ", out)
paste0("
", out, "
")
}
film_desc <- function(x) {
glue::glue_data(x, "
{title}
Released: {release_date}
Director: {director}
{crawl_html(opening_crawl)}
")
}
films <- repurrrsive::sw_films
films <- films[order(sapply(films, "[[", "episode_id"))]
descs <- vapply(films, film_desc, character(1))
writeLines(descs)
rvest/inst/doc/starwars.Rmd 0000644 0001762 0000144 00000001701 15051552265 015456 0 ustar ligges users ---
title: "Star Wars films (static HTML)"
output: rmarkdown::html_vignette
vignette: >
%\VignetteIndexEntry{Star Wars films (static HTML)}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
---
This vignette contains some data about the Star Wars films for use in rvest examples and vignettes.
```{r, echo = FALSE, results = "asis"}
library(rvest)
crawl_html <- function(x) {
out <- gsub("\r", "", x)
out <- gsub("\n\n", "", out)
out <- gsub("\n", " ", out)
paste0("
", out, "
")
}
film_desc <- function(x) {
glue::glue_data(x, "
{title}
Released: {release_date}
Director: {director}
{crawl_html(opening_crawl)}
")
}
films <- repurrrsive::sw_films
films <- films[order(sapply(films, "[[", "episode_id"))]
descs <- vapply(films, film_desc, character(1))
writeLines(descs)
```
rvest/inst/doc/starwars-dynamic.html 0000644 0001762 0000144 00000023773 15054321704 017332 0 ustar ligges users
Star Wars films (dynamic HTML)
Star Wars films (dynamic HTML)
This vignette contains some dynamic generated data about the Star
Wars films for use in rvest examples and vignettes.
rvest/inst/doc/rvest.R 0000644 0001762 0000144 00000010153 15054321704 014426 0 ustar ligges users ## ----echo=FALSE---------------------------------------------------------------
knitr::opts_chunk$set(comment = "#>", collapse = TRUE)
## -----------------------------------------------------------------------------
library(rvest)
## -----------------------------------------------------------------------------
html <- read_html("http://rvest.tidyverse.org/")
class(html)
## -----------------------------------------------------------------------------
html <- minimal_html("
This is a paragraph
")
html
## -----------------------------------------------------------------------------
html <- minimal_html("
This is a heading
This is a paragraph
This is an important paragraph
")
## -----------------------------------------------------------------------------
html |> html_element("h1")
html |> html_elements("p")
html |> html_elements(".important")
html |> html_elements("#first")
## -----------------------------------------------------------------------------
html <- minimal_html("
apple & pear
banana
pineapple
")
html |>
html_elements("li") |>
html_text2()
## -----------------------------------------------------------------------------
html |>
html_elements("li") |>
html_text()
## -----------------------------------------------------------------------------
html <- minimal_html("
This is
a
paragraph.
This is another paragraph.
It has two sentences.
")
## -----------------------------------------------------------------------------
html |>
html_element("body") |>
html_text2() |>
cat()
## -----------------------------------------------------------------------------
html |>
html_element("body") |>
html_text() |>
cat()
## -----------------------------------------------------------------------------
html <- minimal_html("
cats
x
y
1.5
2.7
4.9
1.3
7.2
8.1
")
## -----------------------------------------------------------------------------
html |>
html_node("table") |>
html_table()
## -----------------------------------------------------------------------------
html <- minimal_html("
C-3PO is a droid that weighs 167 kg R2-D2 is a droid that weighs 96 kg Yoda weighs 66 kg R4-P17 is a droid
")
## -----------------------------------------------------------------------------
html |> html_elements("b") |> html_text2()
html |> html_elements("i") |> html_text2()
html |> html_elements(".weight") |> html_text2()
## -----------------------------------------------------------------------------
characters <- html |> html_elements("li")
characters |> html_element("b") |> html_text2()
characters |> html_element("i") |> html_text2()
characters |> html_element(".weight") |> html_text2()
## -----------------------------------------------------------------------------
data.frame(
name = characters |> html_element("b") |> html_text2(),
species = characters |> html_element("i") |> html_text2(),
weight = characters |> html_element(".weight") |> html_text2()
)
rvest/inst/doc/starwars-dynamic.R 0000644 0001762 0000144 00000001141 15054321704 016550 0 ustar ligges users ## ----eval = FALSE, echo = FALSE-----------------------------------------------
# library(magrittr)
# crawl_html <- function(x) {
# x |>
# gsub("\r", "", .) |>
# gsub("\n\n", "", .) |>
# gsub("\n", " ", .) |>
# paste0("
", ., "
")
# }
#
# fields <- c("episode_id", "title", "release_date", "director", "opening_crawl")
# repurrrsive::sw_films |>
# lapply(\(film) film[fields]) |>
# lapply(function(film) {
# film$opening_crawl <- crawl_html(film$opening_crawl)
# film
# }) |>
# jsonlite::write_json("vignettes/starwars.json", pretty = TRUE, auto_unbox = TRUE)
rvest/inst/doc/starwars.html 0000644 0001762 0000144 00000022717 15054321705 015706 0 ustar ligges users
Star Wars films (static HTML)
Star Wars films (static HTML)
This vignette contains some data about the Star Wars films for use in
rvest examples and vignettes.
The Phantom Menace
Released: 1999-05-19
Director: George Lucas
Turmoil has engulfed the Galactic Republic. The taxation of trade routes
to outlying star systems is in dispute.
Hoping to resolve the matter with a blockade of deadly battleships, the
greedy Trade Federation has stopped all shipping to the small planet of
Naboo.
While the Congress of the Republic endlessly debates this alarming chain
of events, the Supreme Chancellor has secretly dispatched two Jedi
Knights, the guardians of peace and justice in the galaxy, to settle the
conflict….
Attack of the Clones
Released: 2002-05-16
Director: George Lucas
There is unrest in the Galactic Senate. Several thousand solar systems
have declared their intentions to leave the Republic.
This separatist movement, under the leadership of the mysterious Count
Dooku, has made it difficult for the limited number of Jedi Knights to
maintain peace and order in the galaxy.
Senator Amidala, the former Queen of Naboo, is returning to the Galactic
Senate to vote on the critical issue of creating an ARMY OF THE REPUBLIC
to assist the overwhelmed Jedi….
Revenge of the Sith
Released: 2005-05-19
Director: George Lucas
War! The Republic is crumbling under attacks by the ruthless Sith Lord,
Count Dooku. There are heroes on both sides. Evil is everywhere.
In a stunning move, the fiendish droid leader, General Grievous, has
swept into the Republic capital and kidnapped Chancellor Palpatine,
leader of the Galactic Senate.
As the Separatist Droid Army attempts to flee the besieged capital with
their valuable hostage, two Jedi Knights lead a desperate mission to
rescue the captive Chancellor….
A New Hope
Released: 1977-05-25
Director: George Lucas
It is a period of civil war. Rebel spaceships, striking from a hidden
base, have won their first victory against the evil Galactic Empire.
During the battle, Rebel spies managed to steal secret plans to the
Empire’s ultimate weapon, the DEATH STAR, an armored space station with
enough power to destroy an entire planet.
Pursued by the Empire’s sinister agents, Princess Leia races home aboard
her starship, custodian of the stolen plans that can save her people and
restore freedom to the galaxy….
The Empire Strikes Back
Released: 1980-05-17
Director: Irvin Kershner
It is a dark time for the Rebellion. Although the Death Star has been
destroyed, Imperial troops have driven the Rebel forces from their
hidden base and pursued them across the galaxy.
Evading the dreaded Imperial Starfleet, a group of freedom fighters led
by Luke Skywalker has established a new secret base on the remote ice
world of Hoth.
The evil lord Darth Vader, obsessed with finding young Skywalker, has
dispatched thousands of remote probes into the far reaches of space….
Return of the Jedi
Released: 1983-05-25
Director: Richard Marquand
Luke Skywalker has returned to his home planet of Tatooine in an attempt
to rescue his friend Han Solo from the clutches of the vile gangster
Jabba the Hutt.
Little does Luke know that the GALACTIC EMPIRE has secretly begun
construction on a new armored space station even more powerful than the
first dreaded Death Star.
When completed, this ultimate weapon will spell certain doom for the
small band of rebels struggling to restore freedom to the galaxy…
The Force Awakens
Released: 2015-12-11
Director: J. J. Abrams
Luke Skywalker has vanished. In his absence, the sinister FIRST ORDER
has risen from the ashes of the Empire and will not rest until
Skywalker, the last Jedi, has been destroyed. With the support of the
REPUBLIC, General Leia Organa leads a brave RESISTANCE. She is desperate
to find her brother Luke and gain his help in restoring peace and
justice to the galaxy. Leia has sent her most daring pilot on a secret
mission to Jakku, where an old ally has discovered a clue to Luke’s
whereabouts….
rvest/inst/doc/rvest.html 0000644 0001762 0000144 00000127524 15054321704 015204 0 ustar ligges users
Web scraping 101
Web scraping 101
This vignette introduces you to the basics of web scraping with
rvest. You’ll first learn the basics of HTML and how to use CSS
selectors to refer to specific elements, then you’ll learn how to use
rvest functions to get data out of HTML and into R.
HTML basics
HTML stands for “HyperText Markup Language” and looks like this:
< html > < head > < title > Page title</ title > </ head > < body > < h1 id = 'first' > A heading</ h1 > < p > Some text & < b > some bold text.</ b ></ p > < img src = 'myimg.png' width = '100' height = '100' > </ body > HTML has a hierarchical structure formed by elements
which consist of a start tag (e.g. <tag>), optional
attributes (id='first'), an end tag (like
</tag>), and contents (everything in
between the start and end tag).
Since < and > are used for start and
end tags, you can’t write them directly. Instead you have to use the
HTML escapes > (greater than) and
< (less than). And since those escapes use
&, if you want a literal ampersand you have to escape
it as &. There are a wide range of possible HTML
escapes but you don’t need to worry about them too much because rvest
automatically handles them for you.
Elements
All up, there are over 100 HTML elements. Some of the most important
are:
Every HTML page must be in an <html> element,
and it must have two children: <head>, which contains
document metadata like the page title, and <body>,
which contains the content you see in the browser.
Block tags like <h1> (heading 1),
<p> (paragraph), and <ol> (ordered
list) form the overall structure of the page.
Inline tags like <b> (bold),
<i> (italics), and <a> (links)
formats text inside block tags.
If you encounter a tag that you’ve never seen before, you can find
out what it does with a little googling. I recommend the MDN Web
Docs which are produced by Mozilla, the company that makes the
Firefox web browser.
Contents
Most elements can have content in between their start and end tags.
This content can either be text or more elements. For example, the
following HTML contains paragraph of text, with one word in bold.
Hi! My name is Hadley.
The children of a node refers only to elements, so
the <p> element above has one child, the
<b> element. The <b> element has
no children, but it does have contents (the text “name”).
Some elements, like <img> can’t have children.
These elements depend solely on attributes for their behavior.
Attributes
Tags can have named attributes which look like
name1='value1' name2='value2'. Two of the most important
attributes are id and class, which are used in
conjunction with CSS (Cascading Style Sheets) to control the visual
appearance of the page. These are often useful when scraping data off a
page.
Reading HTML with rvest
You’ll usually start the scraping process with
read_html(). This returns a xml_document object
which you’ll then manipulate using rvest functions:
<- read_html ("http://rvest.tidyverse.org/" )class (html)#> [1] "xml_document" "xml_node" For examples and experimentation, rvest also includes a function that
lets you create an xml_document from literal HTML:
<- minimal_html (" <p>This is a paragraph<p> <ul> <li>This is a bulleted list</li> </ul> " )#> {html_document} #> <html> #> [1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8 ... #> [2] <body>\n<p>This is a paragraph</p>\n<p>\n </p>\n<ul>\n<li>This is a bull ... Regardless of how you get the HTML, you’ll need some way to identify
the elements that contain the data you care about. rvest provides two
options: CSS selectors and XPath expressions. Here I’ll focus on CSS
selectors because they’re simpler but still sufficiently powerful for
most scraping tasks.
CSS selectors
CSS is short for cascading style sheets, and is a tool for defining
the visual styling of HTML documents. CSS includes a miniature language
for selecting elements on a page called CSS selectors .
CSS selectors define patterns for locating HTML elements, and are useful
for scraping because they provide a concise way of describing which
elements you want to extract.
CSS selectors can be quite complex, but fortunately you only need the
simplest for rvest, because you can also write R code for more
complicated situations. The four most important selectors are:
p: selects all <p>
elements.
.title: selects all elements with class
“title”.
p.special: selects all <p>
elements with class “special”.
#title: selects the element with the id
attribute that equals “title”. Id attributes must be unique within a
document, so this will only ever select a single element.
If you want to learn more CSS selectors I recommend starting with the
fun CSS dinner tutorial and
then referring to the MDN
web docs .
Lets try out the most important selectors with a simple example:
<- minimal_html (" <h1>This is a heading</h1> <p id='first'>This is a paragraph</p> <p class='important'>This is an important paragraph</p> " )In rvest you can extract a single element with
html_element() or all matching elements with
html_elements(). Both functions take a document and a css
selector:
|> html_element ("h1" )#> {html_node} #> <h1> |> html_elements ("p" )#> {xml_nodeset (2)} #> [1] <p id="first">This is a paragraph</p> #> [2] <p class="important">This is an important paragraph</p> |> html_elements (".important" )#> {xml_nodeset (1)} #> [1] <p class="important">This is an important paragraph</p> |> html_elements ("#first" )#> {xml_nodeset (1)} #> [1] <p id="first">This is a paragraph</p> Selectors can also be combined in various ways using
combinators . For example,The most important combinator
is ” “, the descendant combination, because
p a selects all <a> elements that are a
child of a <p> element.
If you don’t know exactly what selector you need, I highly recommend
using SelectorGadget ,
which lets you automatically generate the selector you need by supplying
positive and negative examples in the browser.
Element vs elements
When using rvest, your eventual goal is usually to build up a data
frame, and you want each row to correspond some repeated unit on the
HTML page. In this case, you should generally start by using
html_elements() to select the elements that contain each
observation then use html_element() to extract the
variables from each observation. This guarantees that you’ll get the
same number of values for each variable because
html_element() always returns the same number of outputs as
inputs.
To illustrate this problem take a look at this simple example I
constructed using a few entries from dplyr::starwars:
<- minimal_html (" <ul> <li><b>C-3PO</b> is a <i>droid</i> that weighs <span class='weight'>167 kg</span></li> <li><b>R2-D2</b> is a <i>droid</i> that weighs <span class='weight'>96 kg</span></li> <li><b>Yoda</b> weighs <span class='weight'>66 kg</span></li> <li><b>R4-P17</b> is a <i>droid</i></li> </ul> " )If you try to extract name, species, and weight directly, you end up
with one vector of length four and two vectors of length three, and no
way to align them:
|> html_elements ("b" ) |> html_text2 ()#> [1] "C-3PO" "R2-D2" "Yoda" "R4-P17" |> html_elements ("i" ) |> html_text2 ()#> [1] "droid" "droid" "droid" |> html_elements (".weight" ) |> html_text2 ()#> [1] "167 kg" "96 kg" "66 kg" Instead, use html_elements() to find a element that
corresponds to each character, then use html_element() to
extract each variable for all observations:
<- html |> html_elements ("li" )|> html_element ("b" ) |> html_text2 ()#> [1] "C-3PO" "R2-D2" "Yoda" "R4-P17" |> html_element ("i" ) |> html_text2 ()#> [1] "droid" "droid" NA "droid" |> html_element (".weight" ) |> html_text2 ()#> [1] "167 kg" "96 kg" "66 kg" NA html_element() automatically fills in NA
when no elements match, keeping all of the variables aligned and making
it easy to create a data frame:
data.frame (name = characters |> html_element ("b" ) |> html_text2 (),species = characters |> html_element ("i" ) |> html_text2 (),weight = characters |> html_element (".weight" ) |> html_text2 ()#> name species weight #> 1 C-3PO droid 167 kg #> 2 R2-D2 droid 96 kg #> 3 Yoda <NA> 66 kg #> 4 R4-P17 droid <NA>
rvest/inst/WORDLIST 0000644 0001762 0000144 00000000516 15051434444 013572 0 ustar ligges users CDATA
CMD
Codecov
DTD
HyperText
IMDB
JS
LiveHTML
MDN
PBC
PSVI
RoboBrowser
SelectorGadget
XINCLUDE
XInclude
XPath
arounds
bookmarklet
chromote
colspan
colspans
combinator
combinators
css
fixup
funder
hardcoded
httr
innerText
ith
javascript
libxml
magrittr
nodeset
rowspan
rowspans
selendir
starwars
tibble
tibbles
tidyverse
uris
xpath
rvest/README.md 0000644 0001762 0000144 00000010574 15054317663 012715 0 ustar ligges users
# rvest corresponds
# to a different film
films <- starwars |> html_elements("section")
films
#> {xml_nodeset (7)}
#> [1] \nThe Phantom Menace\n \n\nReleased: 1999 ...
#> [2] \nAttack of the Clones\n \n\nReleased: 20 ...
#> [3] \nRevenge of the Sith\n \n\nReleased: 200 ...
#> [4] \nA New Hope\n \n\nReleased: 1977-05-25\n ...
#> [5] \nThe Empire Strikes Back\n \n\nReleased: ...
#> [6] \nReturn of the Jedi\n \n\nReleased: 1983 ...
#> [7] \nThe Force Awakens\n \n\nReleased: 2015- ...
# Then use html_element() to extract one element per film. Here
# we the title is given by the text inside
title <- films |>
html_element("h2") |>
html_text2()
title
#> [1] "The Phantom Menace" "Attack of the Clones"
#> [3] "Revenge of the Sith" "A New Hope"
#> [5] "The Empire Strikes Back" "Return of the Jedi"
#> [7] "The Force Awakens"
# Or use html_attr() to get data out of attributes. html_attr() always
# returns a string so we convert it to an integer using a readr function
episode <- films |>
html_element("h2") |>
html_attr("data-id") |>
readr::parse_integer()
episode
#> [1] 1 2 3 4 5 6 7
```
If the page contains tabular data you can convert it directly to a data
frame with `html_table()`:
``` r
html <- read_html("https://en.wikipedia.org/w/index.php?title=The_Lego_Movie&oldid=998422565")
html |>
html_element(".tracklist") |>
html_table()
#> # A tibble: 29 × 4
#> No. Title `Performer(s)` Length
#>
#> 1 1. "\"Everything Is Awesome\"" "Tegan and Sara featuring The Lonel… 2:43
#> 2 2. "\"Prologue\"" "" 2:28
#> 3 3. "\"Emmett's Morning\"" "" 2:00
#> 4 4. "\"Emmett Falls in Love\"" "" 1:11
#> 5 5. "\"Escape\"" "" 3:26
#> 6 6. "\"Into the Old West\"" "" 1:00
#> 7 7. "\"Wyldstyle Explains\"" "" 1:21
#> 8 8. "\"Emmett's Mind\"" "" 2:17
#> 9 9. "\"The Transformation\"" "" 1:46
#> 10 10. "\"Saloons and Wagons\"" "" 3:38
#> # ℹ 19 more rows
```
rvest/build/ 0000755 0001762 0000144 00000000000 15054321705 012516 5 ustar ligges users rvest/build/vignette.rds 0000644 0001762 0000144 00000000432 15054321705 015054 0 ustar ligges users RMk@]MHPj `AJ&.$Qvom l4!̼r !UbנZpPuāӑ\i+
)d¤ǘE¿L/B%_SKvN^&CD7" .}]|p_+K李BdKoU(TWج#iDB X^+&[c<?&{ h}D~ȔIdH7`y NA ^ rvest/man/ 0000755 0001762 0000144 00000000000 15051552265 012176 5 ustar ligges users rvest/man/rename.Rd 0000644 0001762 0000144 00000003256 14014035320 013724 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/rename.R
\name{rename}
\alias{set_values}
\alias{submit_form}
\alias{xml_tag}
\alias{xml_node}
\alias{xml_nodes}
\alias{html_nodes}
\alias{html_node}
\alias{back}
\alias{forward}
\alias{jump_to}
\alias{follow_link}
\alias{html_session}
\title{Functions renamed in rvest 1.0.0}
\usage{
set_values(form, ...)
submit_form(session, form, submit = NULL, ...)
xml_tag(x)
xml_node(...)
xml_nodes(...)
html_nodes(...)
html_node(...)
back(x)
forward(x)
jump_to(x, url, ...)
follow_link(x, ...)
html_session(url, ...)
}
\description{
\ifelse{html}{\href{https://lifecycle.r-lib.org/articles/stages.html#deprecated}{\figure{lifecycle-deprecated.svg}{options: alt='[Deprecated]'}}}{\strong{[Deprecated]}}
rvest 1.0.0 renamed a number of functions to ensure that every function
has a common prefix, matching tidyverse conventions that emerged since
rvest was first created.
\itemize{
\item \code{set_values()} -> \code{html_form_set()}
\item \code{submit_form()} -> \code{session_submit()}
\item \code{xml_tag()} -> \code{html_name()}
\item \code{xml_node()} & \code{html_node()} -> \code{html_element()}
\item \code{xml_nodes()} & \code{html_nodes()} -> \code{html_elements()}
}
(\code{html_node()} and \code{html_nodes()} are only superseded because they're
so widely used.)
Additionally all session related functions gained a common prefix:
\itemize{
\item \code{html_session()} -> \code{session()}
\item \code{forward()} -> \code{session_forward()}
\item \code{back()} -> \code{session_back()}
\item \code{jump_to()} -> \code{session_jump_to()}
\item \code{follow_link()} -> \code{session_follow_link()}
}
}
\keyword{internal}
rvest/man/google_form.Rd 0000644 0001762 0000144 00000000470 14101012310 014736 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/utils.R
\name{google_form}
\alias{google_form}
\title{Make link to google form given id}
\usage{
google_form(x)
}
\arguments{
\item{x}{Unique identifier for form}
}
\description{
Make link to google form given id
}
\keyword{internal}
rvest/man/read_html.Rd 0000644 0001762 0000144 00000006542 15051552265 014433 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/rvest-package.R
\name{read_html}
\alias{read_html}
\title{Static web scraping (with xml2)}
\usage{
read_html(
x,
encoding = "",
...,
options = c("RECOVER", "NOERROR", "NOBLANKS", "HUGE")
)
}
\arguments{
\item{x}{Usually a string representing a URL. See \code{\link[xml2:read_xml]{xml2::read_html()}} for
other options.}
\item{encoding}{Specify a default encoding for the document. Unless
otherwise specified XML documents are assumed to be in UTF-8 or
UTF-16. If the document is not UTF-8/16, and lacks an explicit
encoding directive, this allows you to supply a default.}
\item{...}{Additional arguments passed on to methods.}
\item{options}{Set parsing options for the libxml2 parser. Zero or more of
\describe{
\item{RECOVER}{recover on errors}
\item{NOENT}{substitute entities}
\item{DTDLOAD}{load the external subset}
\item{DTDATTR}{default DTD attributes}
\item{DTDVALID}{validate with the DTD}
\item{NOERROR}{suppress error reports}
\item{NOWARNING}{suppress warning reports}
\item{PEDANTIC}{pedantic error reporting}
\item{NOBLANKS}{remove blank nodes}
\item{SAX1}{use the SAX1 interface internally}
\item{XINCLUDE}{Implement XInclude substitution}
\item{NONET}{Forbid network access}
\item{NODICT}{Do not reuse the context dictionary}
\item{NSCLEAN}{remove redundant namespaces declarations}
\item{NOCDATA}{merge CDATA as text nodes}
\item{NOXINCNODE}{do not generate XINCLUDE START/END nodes}
\item{COMPACT}{compact small text nodes; no modification of the tree allowed afterwards (will possibly crash if you try to modify the tree)}
\item{OLD10}{parse using XML-1.0 before update 5}
\item{NOBASEFIX}{do not fixup XINCLUDE xml:base uris}
\item{HUGE}{relax any hardcoded limit from the parser}
\item{OLDSAX}{parse using SAX2 interface before 2.7.0}
\item{IGNORE_ENC}{ignore internal document encoding hint}
\item{BIG_LINES}{Store big lines numbers in text PSVI field}
}}
}
\description{
\code{\link[=read_html]{read_html()}} works by performing a HTTP request then parsing the HTML
received using the xml2 package. This is "static" scraping because it
operates only on the raw HTML file. While this works for most sites,
in some cases you will need to use \code{\link[=read_html_live]{read_html_live()}} if the parts of
the page you want to scrape are dynamically generated with javascript.
Generally, we recommend using \code{read_html()} if it works, as it will be
faster and more robust, as it has fewer external dependencies (i.e. it
doesn't rely on the Chrome web browser installed on your computer.)
}
\examples{
# Start by reading a HTML page with read_html():
starwars <- read_html("https://rvest.tidyverse.org/articles/starwars.html")
# Then find elements that match a css selector or XPath expression
# using html_elements(). In this example, each corresponds
# to a different film
films <- starwars |> html_elements("section")
films
# Then use html_element() to extract one element per film. Here
# we the title is given by the text inside
title <- films |>
html_element("h2") |>
html_text2()
title
# Or use html_attr() to get data out of attributes. html_attr() always
# returns a string so we convert it to an integer using a readr function
episode <- films |>
html_element("h2") |>
html_attr("data-id") |>
readr::parse_integer()
episode
}
rvest/man/session.Rd 0000644 0001762 0000144 00000004667 15051552265 014165 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/session.R
\name{session}
\alias{session}
\alias{is.session}
\alias{session_jump_to}
\alias{session_follow_link}
\alias{session_back}
\alias{session_forward}
\alias{session_history}
\alias{session_submit}
\title{Simulate a session in web browser}
\usage{
session(url, ...)
is.session(x)
session_jump_to(x, url, ...)
session_follow_link(x, i, css, xpath, ...)
session_back(x)
session_forward(x)
session_history(x)
session_submit(x, form, submit = NULL, ...)
}
\arguments{
\item{url}{A URL, either relative or absolute, to navigate to.}
\item{...}{Any additional httr config to use throughout the session.}
\item{x}{A session.}
\item{i}{A integer to select the ith link or a string to match the
first link containing that text (case sensitive).}
\item{css, xpath}{Elements to select. Supply one of \code{css} or \code{xpath}
depending on whether you want to use a CSS selector or XPath 1.0
expression.}
\item{form}{An \link{html_form} to submit}
\item{submit}{Which button should be used to submit the form?
\itemize{
\item \code{NULL}, the default, uses the first button.
\item A string selects a button by its name.
\item A number selects a button using its relative position.
}}
}
\description{
This set of functions allows you to simulate a user interacting with a
website, using forms and navigating from page to page.
\itemize{
\item Create a session with \code{session(url)}
\item Navigate to a specified url with \code{session_jump_to()}, or follow a link on the
page with \code{session_follow_link()}.
\item Submit an \link{html_form} with \code{session_submit()}.
\item View the history with \code{session_history()} and navigate back and forward
with \code{session_back()} and \code{session_forward()}.
\item Extract page contents with \code{\link[=html_element]{html_element()}} and \code{\link[=html_elements]{html_elements()}}, or get the
complete HTML document with \code{\link[=read_html]{read_html()}}.
\item Inspect the HTTP response with \code{\link[httr:cookies]{httr::cookies()}}, \code{\link[httr:headers]{httr::headers()}},
and \code{\link[httr:status_code]{httr::status_code()}}.
}
}
\examples{
s <- session("http://hadley.nz")
s |>
session_jump_to("hadley.jpg") |>
session_jump_to("/") |>
session_history()
s |>
session_jump_to("hadley.jpg") |>
session_back() |>
session_history()
\donttest{
s |>
session_follow_link(css = "p a") |>
html_elements("p")
}
}
rvest/man/html_form.Rd 0000644 0001762 0000144 00000004253 15051552265 014460 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/form.R
\name{html_form}
\alias{html_form}
\alias{html_form_set}
\alias{html_form_submit}
\title{Parse forms and set values}
\usage{
html_form(x, base_url = NULL)
html_form_set(form, ...)
html_form_submit(form, submit = NULL)
}
\arguments{
\item{x}{A document (from \code{\link[=read_html]{read_html()}}), node set (from \code{\link[=html_elements]{html_elements()}}),
node (from \code{\link[=html_element]{html_element()}}), or session (from \code{\link[=session]{session()}}).}
\item{base_url}{Base url of underlying HTML document. The default, \code{NULL},
uses the url of the HTML document underlying \code{x}.}
\item{form}{A form}
\item{...}{<\code{\link[rlang:dyn-dots]{dynamic-dots}}> Name-value pairs giving
fields to modify.
Provide a character vector to set multiple checkboxes in a set or
select multiple values from a multi-select.}
\item{submit}{Which button should be used to submit the form?
\itemize{
\item \code{NULL}, the default, uses the first button.
\item A string selects a button by its name.
\item A number selects a button using its relative position.
}}
}
\value{
\itemize{
\item \code{html_form()} returns as S3 object with class \code{rvest_form} when applied
to a single element. It returns a list of \code{rvest_form} objects when
applied to multiple elements or a document.
\item \code{html_form_set()} returns an \code{rvest_form} object.
\item \code{html_form_submit()} submits the form, returning an httr response which
can be parsed with \code{\link[=read_html]{read_html()}}.
}
}
\description{
Use \code{html_form()} to extract a form, set values with \code{html_form_set()},
and submit it with \code{html_form_submit()}.
}
\examples{
html <- read_html("http://www.google.com")
search <- html_form(html)[[1]]
search <- search |> html_form_set(q = "My little pony", hl = "fr")
# Or if you have a list of values, use !!!
vals <- list(q = "web scraping", hl = "en")
search <- search |> html_form_set(!!!vals)
# To submit and get result:
\dontrun{
resp <- html_form_submit(search)
read_html(resp)
}
}
\seealso{
HTML 4.01 form specification:
\url{https://www.w3.org/TR/html401/interact/forms.html}
}
rvest/man/repair_encoding.Rd 0000644 0001762 0000144 00000001231 14014035320 015574 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/encoding.R
\name{repair_encoding}
\alias{repair_encoding}
\title{Repair faulty encoding}
\usage{
repair_encoding(x, from = NULL)
}
\arguments{
\item{from}{The encoding that the string is actually in. If \code{NULL},
\code{guess_encoding} will be used.}
}
\description{
\ifelse{html}{\href{https://lifecycle.r-lib.org/articles/stages.html#deprecated}{\figure{lifecycle-deprecated.svg}{options: alt='[Deprecated]'}}}{\strong{[Deprecated]}}
This function has been deprecated because it doesn't work. Instead
re-read the HTML file with correct \code{encoding} argument.
}
\keyword{internal}
rvest/man/html_attr.Rd 0000644 0001762 0000144 00000002373 15051552265 014470 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/html.R
\name{html_attr}
\alias{html_attr}
\alias{html_attrs}
\title{Get element attributes}
\usage{
html_attr(x, name, default = NA_character_)
html_attrs(x)
}
\arguments{
\item{x}{A document (from \code{\link[=read_html]{read_html()}}), node set (from \code{\link[=html_elements]{html_elements()}}),
node (from \code{\link[=html_element]{html_element()}}), or session (from \code{\link[=session]{session()}}).}
\item{name}{Name of attribute to retrieve.}
\item{default}{A string used as a default value when the attribute does
not exist in every element.}
}
\value{
A character vector (for \code{html_attr()}) or list (\code{html_attrs()})
the same length as \code{x}.
}
\description{
\code{html_attr()} gets a single attribute; \code{html_attrs()} gets all attributes.
}
\examples{
html <- minimal_html('')
html |> html_elements("a") |> html_attrs()
html |> html_elements("a") |> html_attr("href")
html |> html_elements("li") |> html_attr("class")
html |> html_elements("li") |> html_attr("class", default = "inactive")
}
rvest/man/rvest-package.Rd 0000644 0001762 0000144 00000001462 15051434445 015223 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/rvest-package.R
\docType{package}
\name{rvest-package}
\alias{rvest}
\alias{rvest-package}
\title{rvest: Easily Harvest (Scrape) Web Pages}
\description{
\if{html}{\figure{logo.png}{options: style='float: right' alt='logo' width='120'}}
Wrappers around the 'xml2' and 'httr' packages to make it easy to download, then manipulate, HTML and XML.
}
\seealso{
Useful links:
\itemize{
\item \url{https://rvest.tidyverse.org/}
\item \url{https://github.com/tidyverse/rvest}
\item Report bugs at \url{https://github.com/tidyverse/rvest/issues}
}
}
\author{
\strong{Maintainer}: Hadley Wickham \email{hadley@posit.co}
Other contributors:
\itemize{
\item Posit Software, PBC (03wc8by49) [copyright holder, funder]
}
}
\keyword{internal}
rvest/man/html_text.Rd 0000644 0001762 0000144 00000005010 15051552265 014471 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/text.R
\name{html_text}
\alias{html_text}
\alias{html_text2}
\title{Get element text}
\usage{
html_text(x, trim = FALSE)
html_text2(x, preserve_nbsp = FALSE)
}
\arguments{
\item{x}{A document, node, or node set.}
\item{trim}{If \code{TRUE} will trim leading and trailing spaces.}
\item{preserve_nbsp}{Should non-breaking spaces be preserved? By default,
\code{html_text2()} converts to ordinary spaces to ease further computation.
When \code{preserve_nbsp} is \code{TRUE}, \verb{ } will appear in strings as
\code{"\\ua0"}. This often causes confusion because it prints the same way as
\code{" "}.}
}
\value{
A character vector the same length as \code{x}
}
\description{
There are two ways to retrieve text from a element: \code{html_text()} and
\code{html_text2()}. \code{html_text()} is a thin wrapper around \code{\link[xml2:xml_text]{xml2::xml_text()}}
which returns just the raw underlying text. \code{html_text2()} simulates how
text looks in a browser, using an approach inspired by JavaScript's
\href{https://developer.mozilla.org/en-US/docs/Web/API/HTMLElement/innerText}{innerText()}.
Roughly speaking, it converts \verb{ } tags, and lightly formats tabular data.
\code{html_text2()} is usually what you want, but it is much slower than
\code{html_text()} so for simple applications where performance is important
you may want to use \code{html_text()} instead.
}
\examples{
# To understand the difference between html_text() and html_text2()
# take the following html:
html <- minimal_html(
"
This is a paragraph.
This another sentence.
x y
")
x1 <- html |> html_element("p") |> html_text()
x2 <- html |> html_element("p") |> html_text2()
# When printed, non-breaking spaces look exactly like regular spaces
x1
x2
# But aren't actually the same:
x1 == x2
# Which you can confirm by looking at their underlying binary
# representaion:
charToRaw(x1)
charToRaw(x2)
}
rvest/man/reexports.Rd 0000644 0001762 0000144 00000000735 14557172067 014535 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/rvest-package.R
\docType{import}
\name{reexports}
\alias{reexports}
\alias{url_absolute}
\alias{\%>\%}
\title{Objects exported from other packages}
\keyword{internal}
\description{
These objects are imported from other packages. Follow the links
below to see their documentation.
\describe{
\item{magrittr}{\code{\link[magrittr:pipe]{\%>\%}}}
\item{xml2}{\code{\link[xml2]{url_absolute}}}
}}
rvest/man/figures/ 0000755 0001762 0000144 00000000000 14554031036 013635 5 ustar ligges users rvest/man/figures/lifecycle-questioning.svg 0000644 0001762 0000144 00000002444 14554031036 020664 0 ustar ligges users
lifecycle: questioning
lifecycle
lifecycle
questioning
questioning
rvest/man/figures/lifecycle-stable.svg 0000644 0001762 0000144 00000002472 14554031036 017572 0 ustar ligges users
lifecycle: stable
lifecycle
lifecycle
stable
stable
rvest/man/figures/lifecycle-experimental.svg 0000644 0001762 0000144 00000002450 14554031036 021011 0 ustar ligges users
lifecycle: experimental
lifecycle
lifecycle
experimental
experimental
rvest/man/figures/lifecycle-deprecated.svg 0000644 0001762 0000144 00000002440 14554031036 020413 0 ustar ligges users
lifecycle: deprecated
lifecycle
lifecycle
deprecated
deprecated
rvest/man/figures/lifecycle-superseded.svg 0000644 0001762 0000144 00000002440 14554031036 020456 0 ustar ligges users
lifecycle: superseded
lifecycle
lifecycle
superseded
superseded
rvest/man/figures/logo.png 0000644 0001762 0000144 00000076112 14554031036 015312 0 ustar ligges users PNG
IHDR ޫh cHRM z& u0 ` : pQ< bKGD tIME
.߰h {IDATxw yNg6GE$H"DIlYmْϾ};ߝ}
,%1Q$H"9lΓCQݳK 6a35==}qKW? gN\Sc . 1w` m_ ^ X':/a n _%GP$"@ȗ2DQCVවP.~ |P:/U | Ew |EQĮCf˫_~ CD4P'%6 ~:os ;+xpN䥍:0jko~
0- S}@$^ʨx MN H 5 N䥈:fĹˀ_ /)։tP'A
yO g;rB
u/ |QC\ ^0FS4
y\Dd ?^'EF 3-ǁDe/q$a
4D L 'IJӞ7KL_ NE v ?| ;+ $CF 㺌s d*?u"/.^DԐ7 UEu ⚺N[`6CѶ9u"NyF
y(o7 F@\CiL&j|,9ǯh d3ss⧀ R-.'EI=\s! ;"/&LD6D7wJp}lqQ' 1]aJ91ƭ]$Am4Q >fȖAl
u3;/2!à#=2Jx: r 岔 33%[։<; < ̐݁s,e~8WhFL$\0
,<ܗ aT||?uQ'`ս% .ۤBaIR0O`IJI\?!]*G|E $%/u"E̯3%Ø.F2V;X@.C~~2{- ։|n,YQCS2 / t8W9Dz01Qq%>ދ 2P'gED
q
d~of>d~]$q+bY=J\)ˬ0%[|luQ' p7d~ f>d~VDpDŹYT1,SJ&KEs2S9T-*s
ԉ<ܙw n;-JVv`GLXhS |.~gq?e%v0FY_+G_ܴ'hDЃ\ ]=)/eV* 2%[< |Eͫm~
0iư ._9ȶ2\3WYad+WL!{ jY&*4Z"Q: Rs |B {N ȧP]A(e?frGήPKOU^D 6 Dv̪Oih%_qQ{*p`>Oy~2@XxYͽ2ۙ9 +[r2m31V*Ηl!dH|Yxս)_j6MY!bqRД̏-!Lc-%J\ʼef-e$[ vb+f;Ϥ3%U2(vv go$%7/gnfJx8W:1Y_Zyʥ-ǔlсKė,au71%kxv90MbqCRieStdD>Oƞ leKg}UfcVe~A{$Fk$ֱtQv/|!ۉ> >,Hi/l- yRIiSތ纨D% 1ߧ(fAsxv ci3liᥔdlbJydF%"e.:g+Q; "512?ä39^f}5?; >Qg<וrZFddkq?43ƨYU% 1u k"jo!ĹsVx@1On~28 \l"/ܮ!o)U =-!۶RMl͏V+ y՟+,1#=4jT3)=\Uwצz# x O&@L!х ѯW f
>\!Pu>{14bPHy~2-NxQx
qMd~3WhDm[<K X.[z5TI%i\gcT`pSjҮ0P,q˨yg3mLV u:ZCR%n J>Y KHӪqUZ< Fug ! O) ,(B ce:8$Lj5?w KΖQWH-%JkyzZ#rKEw0Ŕ_^b`Hq~2-gKY3d~A9Z]ShY
fOr*X.Ǐ_=dU6}B>X |k7N{굦oZv="Ժ*wiK,P$hO*@R;jTsutPbGy8a4U}%ϝ[ffQt3d~EgILpD^:Tmq&>4'ZDt!tC 7
8|,nToGJCiql鿷DI:yWA4k2,+լ: a^ OFIL&k#NE@aY7Vo aC *&A, dm?/QsD4J蠨b|/Am|q LY] }fGxB#
3ڊho\"؎CO烙NzLk< /IxX`@ټSp`dt nbk 0Cފ(y N&H`g3xކ*r]*i (pC
Cd*#!h*>#ax|_ *bW)h{08@S#%C:꺎ՉS*줒NS5~-UrhmfA
lE,&Leb϶1T!^u1krO8W*c "e5AhhN7cuښW v
ly4_h,Nu(wbؙ,F$
'k+.Mv 2džeбV]$xCMz\hr3!WW
tTQO^sѺ:ȥy >Md)c&N@zrЩHI98<2|5/PII yʮK w.ea4$7 } 0L ΆZN`N1>>X&8wЉ~y#].5~!AwG_B);kڲR:$!4A54l aK,"zNaD(KZ)
ikiV
@*A%<>AidØZHgB 75RdHCU`Ҵ};L= VIK-/ϜLiX7e?4 s
߸X%tq^1sBs^+=A%B.Hf9Rb e&X]]1uTOF,F!Eale>ruL zu4c(V|_ -wNEz;ډ\[.Ul؎C-7abhI a Og(# LjDȼ;E߲ [J
(qk opȺuiܲ t]eǥq5Dք
zf'/I`U(:AP_wg×! {Aq@ac #O>t]X R!f
ͷނfY W82Jg%pu1R)]HO2o?;wPC)JN)vu;uҦN9E3MrM*o<1AitH{ap:HR8DWU*4_s5nDqxGW^BFkوגW\.K%#46/ Ahu8a)Gl2r'N_ +<1Acx
&(8A1-DZ<
;wٲت dJ:/}]$y3>G+(O))
a^&f5aB;h+Wj lpKjIit[6# ^ dBvΙ 5?^nyYA3և,qLni@JE 4uۭpD3Mb]*Ҍ 3'v+W"]=aA˗Q觜a&he(tx
. vj*0<
FΏդV ʔ_P[~ƂmĺKr* X8nODA3 sf"AbJ S gr+W 4p[ša:Fmsll?S$%j27&?[e5
q\3 #sq&kגEmƈH =FԲnʕ
bʣcuPsCއ,p8bpnRn| n>q 'd_~B& ى5#D tMCիyYo٩K 8Rya).t)wz dP5*Laa L <چs2$B@筷PGBWjkE:N>O
S )Ȟ:Eerm[߷ ;8p?o};nGG O 2$Jϛ&ʿXMKIs<>A#m3Y&80Hqd4 ɃX :(ttP Ghڲm[<
MyjgC8.~WQE_2Ç1QXPc#7@ercT&Ә8-;wP Rzcۦqb˺ɟ9C <+ȟWkIYMN-y{Ivw#. +
6Ed"pQ;zccD;0" |ƭ[VwN.,hhoklՃlh} eCDIpRUNAK_yvy˖1o #G 56ZxN:[G L- 3 q#><]kux<=-\f.xl4 죪)vꗱmDY0R8Dd2CI >LǍ7h
;Ked2 )aVQ b՝'r%F,"@47QD3
NniD~P, $V`cЭ~d Hqhu TYҶk)1IbS%f5QJ Ljd8uȉEy_2_K0åKX\/bx
= 3[[(nE (^eTi
4m݂
زe~נ:%! @,"팽Ń
ӽUi[PCš!N~;d9L==@b]]Z _:ߦ Mݹ'00ͥ%]lwU*qD Fl4[yYηqMH!w nhT5ědŗ01"mm ]'v*L?@2T&&I^EylLwM&"nmۤ֬Q
b1J4lXOcȳϣB`rhMxV*J6Ͻ@۞ nDilՄ:;i&8$slhEl2"--8P @F 77Ӽm+$VL(qNg <1I-W@
0wt&FM4o{udK3S:v
ŁAd]T+/C[˒Pc
n7#&_xQc/%zB!c) jji6iX Xm-38P=+媠T-_F1&pa
b'MkAnM6ni!Ae2M
@6E%})g-T ?:Pٷ͛&`rTR5ʶ=:n$V {$A\ f,kW(`&NBRa|>z>~q[$юv*,V& jj-oBR1QGBo {aB G!4S1D;V. ; |> ٔU"W,Ç=
P1td'GxSX
(f"Ch$Fw(7jj4:J ]9wl'7<˱)OLmo%@joڝ1c1&'Eܱun4Ӥ[q9ZI F <<*?Úu( |?`ŁCQ
4N*P?;R"$zHB_ѣd=H $νsQV3аi#Gqq``AՔưR)\y\r*_(l |L[~zH'u]ڼ_n+?~lKG\zqkLYa qVXf$7mdWȝ>M]4InDhz;q- )L#֊g۸
7X !CitTQ$4 } yz\Fh1 aG
6.H284NH$uk&4F , W]uNC`!T$CHogUOϫ^Ưگٴgʴ^KUP.t IPct;#38HdՄ[Z(1v_G%GLjm@>!二}mZ* #>GgO '{wQ)-_a%HS}Bmo"stOB%EL=RV|q>% z#8^w_u;BRiYX óm6c{ҲRסB"
sg'nıh+ih sڶPe'O2ȣhun-xmsꤝ H` 9v_z5?;98//IAP ߯WI-﹎J,Q kɞ:R"ʭ-hP FW3>ʓ==DtL$иf5iqJ%x[~I(lPSN.Gil J<#b9v kN}'Obhz..y v^P'@-Bg0_T*|͊od!vjpG;+qK%_zULo| ôhܼԺ< //>ZNLԴFyt|_ 6oBr&u4#GU#T ?4?(Ltc9z ʹpE:nɓ2Y
G#4/;uo`[L$C/^P'Q*}0mo+š!MjF~L2#KE&d}!v_KL!5+hs ³mi*4[GImMc՝g{ߧynikegAM7r|tܤ]ɞ:czxt8>N
{xG e ! MJ ]Gs]
6q?2&H˵<8v :Ljl aYx0f6W'E@D@btz4wKD C#|_Zw 0D3YmUc=r# uJb/cD`1zT"li} z$L|
4Ng0[Z(io'~ Ap+cO>MٓDt
29I|rFFFh ~F#H)1*6n,ROqs@~EfK5];
!x#|?wuW,J&pK%Z "md K ^cy "Ң;-;vӃ<xŗGO?}L_?0lXw7ɞ@ÔiL(z$B+y!P<
p躵6"mm$lF>#G|rẖ4_OU۰]M$?JIm@ix {#arg)qS)k8;^v`+G/} qLqSA3tz0ypݵhyĻl F$B]Hb\R
R=u
U] } 788Dj:|ѣtr32HgpSӐ2#}8v.5lɣGѻ ?AqxUd '3 'w$TlB{HY>tʏ$T(@8V.6^<
˚_{ :AsӞ]y>0FFt8r-WYa!!p2LhwSLI]MIt$\(Vts V'40!}:WK\!Bp>ѤD/bQWWY0 %q:, Z PɫUXlY32:ʡ ar[ EIx0Nb˺ ;NnxR8\M>I|3$ׯC@Zw`ɧ(4j.TO+)w$F+qj9LO Wm[7/qNwD;w`߾DO[k*D4Եj#i= f[C`!X;@4RGk:L3#Յq{I+V@8"h`79-\Q;hi /Dd2@ 89RӐvV{H ǷA& r50}˱}Ӎ Ch"B dhmi8mfɜ=1]GFu$MHsv>32ck\W1H =JҶg76&ﹽHۦz
+@P[D"RJ'u$OP\˪L`
AL X9VIڹIh?3g]rOSx9M(\ A ITU$C0 ŋǐ%5{ۯo;O= e_/2E Ltp,S}kl&kHTDU? ){i;W]N#l4"ϲX` ۶n !ku,^@XBpa92bIZ[[XêU- f ) , 4tA\eMSV4a" GY}ó)K? r5jX,AF<瑦 F>FSSyf.fp jObi"&:b+k*p
-U !
zB8PX
U%MO,4\'\?cS|9 r0T~/:t'$InJccG Auv﹞Cq l 2bj}I
M=# O@iЎ=˩n edF$Ў D*㪊.]'xn2>g-[Ceϳ{dfC |x7] D"A.#А-?!A:x *rIXMCh~ؓx@; VO@R-Jee{d3<_.;'l۶mg|G~ng/
^@Mjة C z0xU0 ۶ عs'B//92~nj`@\FHORJ]Ws_%`~aGy:T;M)D(DR[Bһ7nF?oHcߋ:h[/QfBPй;8uoz_5Ր+%-%/f@ErRGyïg>yDU+P0@8RӫBz9' WMX3\1IZڦur|/UϿȍuC3_hwV$ :4 :tc]4778~Bh$q\aeOw0oymoed*JO t6G՞JR"ކFGQT 1
r#o_ ry6oLcS4V*W~ w$|Uޙxd#Bѓ4toN(b۔KeI54L&u==l6=} wuU Fxs8P*2 S娜>-%E#n
C B&a
l߱J?o~e4Mե Sˠt^ 5 OߡӧOsi `
R) <Zi:u{fijjbU,_Ʀ&tàX,bW*\!3<4/LOOmƶh,J8 4r?/}/?ǟz? 8+%* Fn'k( =rqXr%N"aW*ib+{zHRxGzragxxqb455111@ww7]]h~?xuCJyA2|s}[?|ChWBNK m)g.>sG$!aBs)Xlh,FOOOu'ri*
S,TiPň~YqaSO?×ؖ/p aP~1?~
Mυt,/Bnbe _/ O=G?lܰv,bppt:8tttǫ{366F"@5*
B4B!LDJIX"˒N HE, gtdo~k>8?94ʺe0<BAvӎΣi7~w~ü˲A0TJ $lCױ
|q4P('=
+{Vi\8qo[pͬ?Ԛ^mWA/!xB CVj YS]!
d(^*M?Wod ˅44Vo;?vcin>LChljap,
rK Y`@Ž3M{Tl,jn
>e;=+059߯C]NxAHscSr ~ΫƜ\1g{ϳ|ϸ .uqqT1:XtHvMbQUGd>OBu,*}ɟ MЛ(??>KuױB
DozL4gz!G ^5/[|dOLmk_M|ᬯ}Lau+YLMs\ezͮ |H)UYLMJCŰ֭\Ճ?@!1KU sU|aY4՛4c.[ю"=or /_PrBGq.dx,N n&0|#J1cbYPUf6JA炔 R CC Gh[o~h$BӘ>EwѢp3&$8'<7+j}duR&J:k"w~Ʋ.hTIƝLS>xHp'&_"7;FoZ-Pۗ
vpYS(>a$;YMYI~
W}췾οi|`wlhk@Dp 3 w>4%BqGF0Zozڸ7M5! wq/cS㿙O^Eoi@C hi~B@(ކF(mE}ȳ8p#~ 1! #ԌLdbo_^k|k~N-'zg*Nm#˥8/{=xk$wJ
{ z ];pg(_uQbXNoFBT GJH- Vin^ U=OqŒrái!?;6
:
u"k];MZO4z^:^S,P#4/T4oWtr=𮝘˺k!{
a =?cb6l&{OZC]o'b 6_$`J)hߚF^/_ { hFk+[|֦0^ F!r=PmY d)}Qg\z+@ϝ :kIr>{@+T 3$W_v5-DXofS)>j
Qo ghCd2-#n8I'(=<Ϳ Tl G>_~zut+O/ ~xsD\G}~'A_'oPA_73b;Zkft | DzSS)bo}BӰ{/N[,s E|"ٔ?7˺ 9pG` y ;"ԌL8t _zG;M8BBx{~T@jzUd5mzl)q'&n5.~ !ғz n6ga-YQe 0"\?}o
]u&y4P9qrBDMx btvT,{ϐ~pYehu'ǍNR!, ciO_S$4zC}sЗA<8;>P,(=@{adzghCL;lmLh {íS@yK*y ֺ5DjJtFYr<#p)}}&?LU/ћHmtݧije][EY>|;Z(FYZ햪EGQSiM P$XӴݽR BUqȟ "zkxڰퟖEc~Q'bB(>Y 1w 4bwVL1xiT !aDoI6oRZ>*Qz RӎmY^j﹏NlzCù~v:u/2ԲL=O҆wnhk#z
U%$wAWቧ|"F&sIVe- xU] أj_4
z&Z"d{~~9K_K$CjkĪ;v沖,U {UX'N!pz(r@K?O7E#J
75"LNyS ]TBx5 JˇfnX?B 2>F[ޢ_9#ǰO C)0I!% mڸ7c ?Ay'p'ٳ6nH{?j|Д>r-֪ ST^9N]OاN_QmuKH*RbAu p'& ERJ?;:z^j'i8 [\ȵ;9V7E%b^DShFB7sʑ_Z33B2/g\mDUݝLE"}l\)[JJ=i$G˨p}Ig\Pus =bx{*c3 )}{{} NT|jus
'~ϟ$ !~Lߟ\DCsK}WJƤa2X0k];<03 cHzfA 0_ȋH H:h@]D^I c Dciٟt%m9 <) w#Y>+;S~kV+RIIXsd!^: GK1t;'|]T p,
;Z[S aj e) g ϲߵ <{ j> &pNܿD]h4Չ\9Hɸ 8 e`,8X mW "}~OA^AH^q W3 )< Qɩ_.j Dgs@e6.4uk\E,֔?:se^SG z"G|=h9TSYX*YĹ幻
|
7}[|UTE_`)
Bi 6E
ya x1_~
hбf;F q0Y
bpp
0ǫ!܄rߊZ Sv[:)
>T`
0\!r x/I`lƪ-elEyxeqL7p`[QPP kzYf s87 ̃o ,&/,ܟA䫀 >M\kU25b9By, Q/q,! D6k2Cuk\yboXz2Gx `a`Q{
SQūf3V]XDž2?U82WUW] E3L.t8YSYP; Γo)ߑb7Eg-~x s-6!h`:y R282BOE'n>k F-
zY="8o$ %[u([ e+3e~q^d~Qy Kj.E(bۅU[H) `Jw 8`I=L#:LbĕU=,d~%=g߁" AZ/˼1S7ǹP~s-ZBвe3e~#cd2?ߧQ2"ԉ;\q5d lx5e3>Y862(ߗe~:DaJycMYvZl"K)r0G(S2:q: 3d-.t]ϔ
I .+DXxqo.YF%&[, 8lP]f3^lC4Γl2?< 9Ź'P2s u/ fy S klSh$,sd~_Ec\2B$[\@}Llw1P'"bl]("fŰ C2ͯ.[d 8lP:az.^˔LpNE 3)bㅎu.b v9fuL]6 \"cldօU+[ 2}%+Cu/EI`lƪs2_ŗՉ4P'B2FK)[%Ϡv(^Oy.P'%"70G.Q'%mq.7 |a w[,2%bvP'%7m1]e~41.7 |`נj?tP=p xb>0Cx+(⟡v.3Y3a tEXtSoftware Adobe ImageReadyqe< IENDB` rvest/man/figures/lifecycle-archived.svg 0000644 0001762 0000144 00000002430 14554031036 020077 0 ustar ligges users
lifecycle: archived
lifecycle
lifecycle
archived
archived
rvest/man/figures/lifecycle-defunct.svg 0000644 0001762 0000144 00000002424 14554031036 017745 0 ustar ligges users
lifecycle: defunct
lifecycle
lifecycle
defunct
defunct
rvest/man/figures/lifecycle-soft-deprecated.svg 0000644 0001762 0000144 00000002466 14554031036 021374 0 ustar ligges users
lifecycle: soft-deprecated
lifecycle
lifecycle
soft-deprecated
soft-deprecated
rvest/man/figures/lifecycle-maturing.svg 0000644 0001762 0000144 00000002430 14554031036 020140 0 ustar ligges users
lifecycle: maturing
lifecycle
lifecycle
maturing
maturing
rvest/man/html_element.Rd 0000644 0001762 0000144 00000006031 15051552265 015142 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/selectors.R
\name{html_element}
\alias{html_element}
\alias{html_elements}
\title{Select elements from an HTML document}
\usage{
html_element(x, css, xpath)
html_elements(x, css, xpath)
}
\arguments{
\item{x}{Either a document, a node set or a single node.}
\item{css, xpath}{Elements to select. Supply one of \code{css} or \code{xpath}
depending on whether you want to use a CSS selector or XPath 1.0
expression.}
}
\value{
\code{html_element()} returns a nodeset the same length as the input.
\code{html_elements()} flattens the output so there's no direct way to map
the output to the input.
}
\description{
\code{html_element()} and \code{html_elements()} find HTML element using CSS selectors
or XPath expressions. CSS selectors are particularly useful in conjunction
with \url{https://selectorgadget.com/}, which makes it very easy to discover the
selector you need.
}
\section{CSS selector support}{
CSS selectors are translated to XPath selectors by the \pkg{selectr}
package, which is a port of the python \pkg{cssselect} library,
\url{https://pythonhosted.org/cssselect/}.
It implements the majority of CSS3 selectors, as described in
\url{https://www.w3.org/TR/2011/REC-css3-selectors-20110929/}. The
exceptions are listed below:
\itemize{
\item Pseudo selectors that require interactivity are ignored:
\verb{:hover}, \verb{:active}, \verb{:focus}, \verb{:target}, \verb{:visited}.
\item The following pseudo classes don't work with the wild card element, *:
\verb{*:first-of-type}, \verb{*:last-of-type}, \verb{*:nth-of-type},
\verb{*:nth-last-of-type}, \verb{*:only-of-type}
\item It supports \verb{:contains(text)}
\item You can use !=, \verb{[foo!=bar]} is the same as \verb{:not([foo=bar])}
\item \verb{:not()} accepts a sequence of simple selectors, not just a single
simple selector.
}
}
\examples{
html <- minimal_html("
This is a heading
This is a paragraph
This is an important paragraph
")
html |> html_element("h1")
html |> html_elements("p")
html |> html_elements(".important")
html |> html_elements("#first")
# html_element() vs html_elements() --------------------------------------
html <- minimal_html("
C-3PO is a droid that weighs 167 kg R2-D2 is a droid that weighs 96 kg Yoda weighs 66 kg R4-P17 is a droid
")
li <- html |> html_elements("li")
# When applied to a node set, html_elements() returns all matching elements
# beneath any of the inputs, flattening results into a new node set.
li |> html_elements("i")
# When applied to a node set, html_element() always returns a vector the
# same length as the input, using a "missing" element where needed.
li |> html_element("i")
# and html_text() and html_attr() will return NA
li |> html_element("i") |> html_text2()
li |> html_element("span") |> html_attr("class")
}
rvest/man/LiveHTML.Rd 0000644 0001762 0000144 00000020266 15054317632 014057 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/live.R
\name{LiveHTML}
\alias{LiveHTML}
\title{Interact with a live web page}
\description{
\ifelse{html}{\href{https://lifecycle.r-lib.org/articles/stages.html#experimental}{\figure{lifecycle-experimental.svg}{options: alt='[Experimental]'}}}{\strong{[Experimental]}}
You construct an LiveHTML object with \code{\link[=read_html_live]{read_html_live()}} and then interact,
like you're a human, using the methods described below. When debugging a
scraping script it is particularly useful to use \verb{$view()}, which will open
a live preview of the site, and you can actually see each of the operations
performed on the real site.
rvest provides relatively simple methods for scrolling, typing, and
clicking. For richer interaction, you probably want to use a package
that exposes a more powerful user interface, like
\href{https://ashbythorpe.github.io/selenider/}{selendir}.
}
\examples{
\dontrun{
# To retrieve data for this paginated site, we need to repeatedly push
# the "Load More" button
sess <- read_html_live("https://www.bodybuilding.com/exercises/finder")
sess$view()
sess |> html_elements(".ExResult-row") |> length()
sess$click(".ExLoadMore-btn")
sess |> html_elements(".ExResult-row") |> length()
sess$click(".ExLoadMore-btn")
sess |> html_elements(".ExResult-row") |> length()
}
}
\section{Public fields}{
\if{html}{\out{}}
\describe{
\item{\code{session}}{Underlying chromote session object. For expert use only.}
}
\if{html}{\out{
}}
}
\section{Methods}{
\subsection{Public methods}{
\itemize{
\item \href{#method-LiveHTML-new}{\code{LiveHTML$new()}}
\item \href{#method-LiveHTML-print}{\code{LiveHTML$print()}}
\item \href{#method-LiveHTML-view}{\code{LiveHTML$view()}}
\item \href{#method-LiveHTML-html_elements}{\code{LiveHTML$html_elements()}}
\item \href{#method-LiveHTML-click}{\code{LiveHTML$click()}}
\item \href{#method-LiveHTML-get_scroll_position}{\code{LiveHTML$get_scroll_position()}}
\item \href{#method-LiveHTML-scroll_into_view}{\code{LiveHTML$scroll_into_view()}}
\item \href{#method-LiveHTML-scroll_to}{\code{LiveHTML$scroll_to()}}
\item \href{#method-LiveHTML-scroll_by}{\code{LiveHTML$scroll_by()}}
\item \href{#method-LiveHTML-type}{\code{LiveHTML$type()}}
\item \href{#method-LiveHTML-press}{\code{LiveHTML$press()}}
\item \href{#method-LiveHTML-clone}{\code{LiveHTML$clone()}}
}
}
\if{html}{\out{}}\preformatted{LiveHTML$new(url)}\if{html}{\out{
}}
}
\subsection{Arguments}{
\if{html}{\out{}}
\describe{
\item{\code{url}}{URL to page.}
}
\if{html}{\out{
}}
}
}
\if{html}{\out{}}\preformatted{LiveHTML$print(...)}\if{html}{\out{
}}
}
\subsection{Arguments}{
\if{html}{\out{}}
\describe{
\item{\code{...}}{Ignored}
}
\if{html}{\out{
}}
}
}
\if{html}{\out{}}\preformatted{LiveHTML$view()}\if{html}{\out{
}}
}
}
\if{html}{\out{}}\preformatted{LiveHTML$html_elements(css, xpath)}\if{html}{\out{
}}
}
\subsection{Arguments}{
\if{html}{\out{}}
\describe{
\item{\code{css, xpath}}{CSS selector or xpath expression.}
}
\if{html}{\out{
}}
}
}
\if{html}{\out{}}\preformatted{LiveHTML$click(css, n_clicks = 1)}\if{html}{\out{
}}
}
\subsection{Arguments}{
\if{html}{\out{}}
\describe{
\item{\code{css}}{CSS selector.}
\item{\code{n_clicks}}{Number of clicks}
}
\if{html}{\out{
}}
}
}
\if{html}{\out{}}\preformatted{LiveHTML$get_scroll_position()}\if{html}{\out{
}}
}
}
\if{html}{\out{}}\preformatted{LiveHTML$scroll_into_view(css)}\if{html}{\out{
}}
}
\subsection{Arguments}{
\if{html}{\out{}}
\describe{
\item{\code{css}}{CSS selector.}
}
\if{html}{\out{
}}
}
}
\if{html}{\out{}}\preformatted{LiveHTML$scroll_to(top = 0, left = 0)}\if{html}{\out{
}}
}
\subsection{Arguments}{
\if{html}{\out{}}
\describe{
\item{\code{top, left}}{Number of pixels from top/left respectively.}
}
\if{html}{\out{
}}
}
}
\if{html}{\out{}}\preformatted{LiveHTML$scroll_by(top = 0, left = 0)}\if{html}{\out{
}}
}
\subsection{Arguments}{
\if{html}{\out{}}
\describe{
\item{\code{top, left}}{Number of pixels to scroll up/down and left/right
respectively.}
}
\if{html}{\out{
}}
}
}
\if{html}{\out{}}\preformatted{LiveHTML$type(css, text)}\if{html}{\out{
}}
}
\subsection{Arguments}{
\if{html}{\out{}}
\describe{
\item{\code{css}}{CSS selector.}
\item{\code{text}}{A single string containing the text to type.}
}
\if{html}{\out{
}}
}
}
\if{html}{\out{}}\preformatted{LiveHTML$press(css, key_code, modifiers = character())}\if{html}{\out{
}}
}
\subsection{Arguments}{
\if{html}{\out{}}
\describe{
\item{\code{css}}{CSS selector.}
\item{\code{key_code}}{Name of key. You can see a complete list of known
keys at \url{https://pptr.dev/api/puppeteer.keyinput}.}
\item{\code{modifiers}}{A character vector of modifiers. Must be one or more
of \verb{"Shift}, \code{"Control"}, \code{"Alt"}, or \code{"Meta"}.}
}
\if{html}{\out{
}}
}
}
\if{html}{\out{}}\preformatted{LiveHTML$clone(deep = FALSE)}\if{html}{\out{
}}
}
\subsection{Arguments}{
\if{html}{\out{}}
\describe{
\item{\code{deep}}{Whether to make a deep clone.}
}
\if{html}{\out{
}}
}
}
}
rvest/man/minimal_html.Rd 0000644 0001762 0000144 00000000663 13775437157 015161 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/utils.R
\name{minimal_html}
\alias{minimal_html}
\title{Create an HTML document from inline HTML}
\usage{
minimal_html(html, title = "")
}
\arguments{
\item{html}{HTML contents of page.}
\item{title}{Page title (required by HTML spec).}
}
\description{
Create an HTML document from inline HTML
}
\examples{
minimal_html("test
")
}
\keyword{internal}
rvest/man/html_encoding_guess.Rd 0000644 0001762 0000144 00000002027 15051552265 016506 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/encoding.R
\name{html_encoding_guess}
\alias{html_encoding_guess}
\alias{guess_encoding}
\title{Guess faulty character encoding}
\usage{
html_encoding_guess(x)
}
\arguments{
\item{x}{A character vector.}
}
\description{
\code{html_encoding_guess()} helps you handle web pages that declare an incorrect
encoding. Use \code{html_encoding_guess()} to generate a list of possible
encodings, then try each out by using \code{encoding} argument of \code{read_html()}.
\code{html_encoding_guess()} replaces the deprecated \code{guess_encoding()}.
}
\examples{
# A file with bad encoding included in the package
path <- system.file("html-ex", "bad-encoding.html", package = "rvest")
x <- read_html(path)
x |> html_elements("p") |> html_text()
html_encoding_guess(x)
# Two valid encodings, only one of which is correct
read_html(path, encoding = "ISO-8859-1") |> html_elements("p") |> html_text()
read_html(path, encoding = "ISO-8859-2") |> html_elements("p") |> html_text()
}
rvest/man/html_name.Rd 0000644 0001762 0000144 00000001245 15051552265 014433 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/html.R
\name{html_name}
\alias{html_name}
\title{Get element name}
\usage{
html_name(x)
}
\arguments{
\item{x}{A document (from \code{\link[=read_html]{read_html()}}), node set (from \code{\link[=html_elements]{html_elements()}}),
node (from \code{\link[=html_element]{html_element()}}), or session (from \code{\link[=session]{session()}}).}
}
\value{
A character vector the same length as \code{x}
}
\description{
Get element name
}
\examples{
url <- "https://rvest.tidyverse.org/articles/starwars.html"
html <- read_html(url)
html |>
html_element("div") |>
html_children() |>
html_name()
}
rvest/man/html_table.Rd 0000644 0001762 0000144 00000004643 15051552265 014607 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/table.R
\name{html_table}
\alias{html_table}
\title{Parse an html table into a data frame}
\usage{
html_table(
x,
header = NA,
trim = TRUE,
fill = deprecated(),
dec = ".",
na.strings = "NA",
convert = TRUE
)
}
\arguments{
\item{x}{A document (from \code{\link[=read_html]{read_html()}}), node set (from \code{\link[=html_elements]{html_elements()}}),
node (from \code{\link[=html_element]{html_element()}}), or session (from \code{\link[=session]{session()}}).}
\item{header}{Use first row as header? If \code{NA}, will use first row
if it consists of \verb{} tags.
If \code{TRUE}, column names are left exactly as they are in the source
document, which may require post-processing to generate a valid data
frame.}
\item{trim}{Remove leading and trailing whitespace within each cell?}
\item{fill}{Deprecated - missing cells in tables are now always
automatically filled with \code{NA}.}
\item{dec}{The character used as decimal place marker.}
\item{na.strings}{Character vector of values that will be converted to \code{NA}
if \code{convert} is \code{TRUE}.}
\item{convert}{If \code{TRUE}, will run \code{\link[=type.convert]{type.convert()}} to interpret texts as
integer, double, or \code{NA}.}
}
\value{
When applied to a single element, \code{html_table()} returns a single tibble.
When applied to multiple elements or a document, \code{html_table()} returns
a list of tibbles.
}
\description{
The algorithm mimics what a browser does, but repeats the values of merged
cells in every cell that cover.
}
\examples{
sample1 <- minimal_html("")
sample1 |>
html_element("table") |>
html_table()
# Values in merged cells will be duplicated
sample2 <- minimal_html("")
sample2 |>
html_element("table") |>
html_table()
# If a row is missing cells, they'll be filled with NAs
sample3 <- minimal_html("")
sample3 |>
html_element("table") |>
html_table()
}
rvest/man/read_html_live.Rd 0000644 0001762 0000144 00000004023 15051552265 015442 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/live.R
\name{read_html_live}
\alias{read_html_live}
\title{Live web scraping (with chromote)}
\usage{
read_html_live(url)
}
\arguments{
\item{url}{Website url to read from.}
}
\value{
\code{read_html_live()} returns an R6 \link{LiveHTML} object. You can interact
with this object using the usual rvest functions, or call its methods,
like \verb{$click()}, \verb{$scroll_to()}, and \verb{$type()} to interact with the live
page like a human would.
}
\description{
\ifelse{html}{\href{https://lifecycle.r-lib.org/articles/stages.html#experimental}{\figure{lifecycle-experimental.svg}{options: alt='[Experimental]'}}}{\strong{[Experimental]}}
\code{\link[=read_html]{read_html()}} operates on the HTML source code downloaded from the server.
This works for most websites but can fail if the site uses javascript to
generate the HTML. \code{read_html_live()} provides an alternative interface
that runs a live web browser (Chrome) in the background. This allows you to
access elements of the HTML page that are generated dynamically by javascript
and to interact with the live page by clicking on buttons or typing in
forms.
Behind the scenes, this function uses the
\href{https://rstudio.github.io/chromote/}{chromote} package, which requires that
you have a copy of \href{https://www.google.com/chrome/}{Google Chrome} installed
on your machine.
}
\examples{
\dontrun{
# When we retrieve the raw HTML for this site, it doesn't contain the
# data we're interested in:
static <- read_html("https://www.forbes.com/top-colleges/")
static |> html_element("table")
# Instead, we need to run the site in a real web browser, causing it to
# download a JSON file and then dynamically generate the html:
dynamic <- read_html_live("https://www.forbes.com/top-colleges/")
# You may need to click the cookie consent banner if it appears
dynamic$view()
# Now we can find the table
dynamic |> html_element("table")
# And extract data from it
dynamic |>
html_element("table") |>
html_table()
}
}
rvest/man/html_children.Rd 0000644 0001762 0000144 00000001262 13776122153 015303 0 ustar ligges users % Generated by roxygen2: do not edit by hand
% Please edit documentation in R/html.R
\name{html_children}
\alias{html_children}
\title{Get element children}
\usage{
html_children(x)
}
\arguments{
\item{x}{A document (from \code{\link[=read_html]{read_html()}}), node set (from \code{\link[=html_elements]{html_elements()}}),
node (from \code{\link[=html_element]{html_element()}}), or session (from \code{\link[=session]{session()}}).}
}
\description{
Get element children
}
\examples{
html <- minimal_html("")
ul <- html_elements(html, "ul")
html_children(ul)
html <- minimal_html("Hello Hadley ! ")
p <- html_elements(html, "p")
html_children(p)
}
rvest/DESCRIPTION 0000644 0001762 0000144 00000002426 15054331142 013125 0 ustar ligges users Package: rvest
Title: Easily Harvest (Scrape) Web Pages
Version: 1.0.5
Authors@R: c(
person("Hadley", "Wickham", , "hadley@posit.co", role = c("aut", "cre")),
person("Posit Software, PBC", role = c("cph", "fnd"),
comment = c(ROR = "03wc8by49"))
)
Description: Wrappers around the 'xml2' and 'httr' packages to make it
easy to download, then manipulate, HTML and XML.
License: MIT + file LICENSE
URL: https://rvest.tidyverse.org/, https://github.com/tidyverse/rvest
BugReports: https://github.com/tidyverse/rvest/issues
Depends: R (>= 4.1)
Imports: cli, glue, httr (>= 0.5), lifecycle (>= 1.0.3), magrittr,
rlang (>= 1.1.0), selectr, tibble, xml2 (>= 1.4.0)
Suggests: chromote, covr, knitr, purrr, R6, readr, repurrrsive,
rmarkdown, spelling, stringi (>= 0.3.1), testthat (>= 3.0.2),
tidyr, webfakes
VignetteBuilder: knitr
Config/Needs/website: tidyverse/tidytemplate
Config/testthat/edition: 3
Config/testthat/parallel: true
Encoding: UTF-8
Language: en-US
RoxygenNote: 7.3.2
NeedsCompilation: no
Packaged: 2025-08-29 12:57:41 UTC; hadleywickham
Author: Hadley Wickham [aut, cre],
Posit Software, PBC [cph, fnd] (ROR: )
Maintainer: Hadley Wickham
Repository: CRAN
Date/Publication: 2025-08-29 14:00:02 UTC