xml-urls

Get all URLs detected inside a Feed/Atom/RSS/Sitemap xml markup.

$ npm install xml-urls --save

const xmlUrls = require('xml-urls')

;(async () => {
  const url = process.argv[2]
  if (!url) throw new TypeError('Need to provide an url as first argument.')
  const urls = await xmlUrls(url)

  urls.forEach(url => console.log(url))

  // => [
  //  'http://www.sitemappro.com/',
  //  'http://www.sitemappro.com/download.html',
  //  'http://www.sitemappro.com/register.html',
  //  'http://www.sitemappro.com/examples.html',
  //  'http://www.sitemappro.com/company.html',
  //  'http://www.sitemappro.com/contact.html',
  //  ...
  // ]
})()

By default, URLs are loaded with html-get. Nested sitemap indexes are walked until every page loc is collected.

Pass markup when you already have the first document; nested .xml locs are still fetched:

const urls = await xmlUrls(xml, { url: 'https://example.com/sitemap.xml' })

In a Microlink Function, pass a fetcher so the isolate HTTP client is used instead of html-get:

const maps = require('robots-parser')(robotsUrl, body).getSitemaps()
return require('xml-urls')(maps, { fetcher: fetch })

See more at examples.

Required
Type: string | string[]

A sitemap/feed URL, XML/HTML markup (a string that starts with <), or a list of them.

Type: object

When input is a URL and no fetcher is set, extra options are forwarded to html-get.

Type: function
Default: html-get

(url, opts) => string | { html } | Response | Promise<…>. Return XML/HTML text, an html-get result, or a Fetch Response (gzip is decoded).

Type: string

Parse this markup instead of fetching input. Nested .xml locs are still fetched. Use it with a URL input (or options.url) so relative locs resolve correctly.

Type: string

Base URL when input is markup.

Type: array
Default: []

A list of links to be excluded from the final output. It supports regex patterns.

See [matcher](https://github.com/sindresorhus/matcher#matcher-= for know more.

• html-urls – Get all urls from a HTML markup.
• css-urls – Get all URLs referenced from stylesheet files.

xml-urls © Kiko Beats, released under the MIT License.
Authored and maintained by Kiko Beats with help from contributors.

kikobeats.com · GitHub @Kiko Beats · X @Kikobeats