the validationlab.eu marketing site: 3 audience pages (en) + legal skeletons, i18n scaffold (25 locales, subdirectory scheme), structured frontmatter blocks, Zoho Flow lead forms with capability tagging + honeypot, sitemap/hreflang/JSON-LD/robots, cookie-gated analytics plumbing

This commit is contained in:
2026-11-21 09:25:06 +01:00
parent 95af7107d0
commit 477bf5114f
1076 changed files with 227743 additions and 110 deletions
+22
View File
@@ -0,0 +1,22 @@
(The MIT License)
Copyright (c) 2011 Eugene Kalinin
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+326
View File
@@ -0,0 +1,326 @@
# sitemap ![MIT License](https://img.shields.io/npm/l/sitemap)[![Build Status](https://github.com/ekalinin/sitemap.js/workflows/Node%20CI/badge.svg)](https://github.com/ekalinin/sitemap.js/actions)![Monthly Downloads](https://img.shields.io/npm/dm/sitemap)
**sitemap** is a high-level streaming sitemap-generating library/CLI that
makes creating [sitemap XML](http://www.sitemaps.org/) files easy. [What is a sitemap?](https://support.google.com/webmasters/answer/156184?hl=en&ref_topic=4581190)
## Table of Contents
- [Installation](#installation)
- [Generate a one time sitemap from a list of urls](#generate-a-one-time-sitemap-from-a-list-of-urls)
- [Example of using sitemap.js with](#serve-a-sitemap-from-a-server-and-periodically-update-it) [express](https://expressjs.com/)
- [Generating more than one sitemap](#create-sitemap-and-index-files-from-one-large-list)
- [Options you can pass](#options-you-can-pass)
- [Examples](#examples)
- [API](#api)
- [Maintainers](#maintainers)
- [License](#license)
## Installation
```sh
npm install --save sitemap
```
Requires Node.js >= 22.22.1. This package is **ESM-only** — it no longer ships a
CommonJS build. Load it with `import`; `require('sitemap')` works only on Node
versions with built-in `require(esm)` support, which needs 22.12 or later;
the minimum is 22.22.1 to match the toolchain used to develop it.
## Generate a one time sitemap from a list of urls
If you are just looking to take a giant list of URLs and turn it into some sitemaps, try out our CLI. The cli can also parse, update and validate existing sitemaps.
```sh
npx sitemap < listofurls.txt # `npx sitemap -h` for more examples and a list of options.
```
For programmatic one time generation of a sitemap try:
```js
import { SitemapStream, streamToPromise } from 'sitemap'
import { Readable } from 'stream'
// An array with your links
const links = [{ url: '/page-1/', changefreq: 'daily', priority: 0.3 }]
// Create a stream to write to
const stream = new SitemapStream( { hostname: 'https://...' } )
// Return a promise that resolves with your XML string
return streamToPromise(Readable.from(links).pipe(stream)).then((data) =>
data.toString()
)
```
## Serve a sitemap from a server and periodically update it
Use this if you have less than 50 thousand urls. See SitemapAndIndexStream for if you have more.
```js
import express from 'express'
import { SitemapStream, streamToPromise } from 'sitemap'
import { createGzip } from 'zlib'
import { Readable } from 'stream'
const app = express()
let sitemap
app.get('/sitemap.xml', function(req, res) {
res.header('Content-Type', 'application/xml');
res.header('Content-Encoding', 'gzip');
// if we have a cached entry send it
if (sitemap) {
res.send(sitemap)
return
}
try {
const smStream = new SitemapStream({ hostname: 'https://example.com/' })
const pipeline = smStream.pipe(createGzip())
// pipe your entries or directly write them.
smStream.write({ url: '/page-1/', changefreq: 'daily', priority: 0.3 })
smStream.write({ url: '/page-2/', changefreq: 'monthly', priority: 0.7 })
smStream.write({ url: '/page-3/'}) // changefreq: 'weekly', priority: 0.5
smStream.write({ url: '/page-4/', img: "http://urlTest.com" })
/* or use
Readable.from([{url: '/page-1'}...]).pipe(smStream)
if you are looking to avoid writing your own loop.
*/
// cache the response
streamToPromise(pipeline).then(sm => sitemap = sm)
// make sure to attach a write stream such as streamToPromise before ending
smStream.end()
// stream write the response
pipeline.pipe(res).on('error', (e) => {throw e})
} catch (e) {
console.error(e)
res.status(500).end()
}
})
app.listen(3000, () => {
console.log('listening')
});
```
## Create sitemap and index files from one large list
If you know you are definitely going to have more than 50,000 urls in your sitemap, you can use this slightly more complex interface to create a new sitemap every 45,000 entries and add that file to a sitemap index.
```js
import { createReadStream, createWriteStream } from 'fs'
import { resolve } from 'path'
import { createGzip } from 'zlib'
import { simpleSitemapAndIndex, lineSeparatedURLsToSitemapOptions } from 'sitemap'
// writes sitemaps and index out to the destination you provide.
simpleSitemapAndIndex({
hostname: 'https://example.com',
destinationDir: './',
sourceData: lineSeparatedURLsToSitemapOptions(
createReadStream('./your-data.json.txt')
),
// sourceData can also be:
// sourceData: [{ url: '/page-1/', changefreq: 'daily'}, ...],
// or
// sourceData: './your-data.json.txt',
limit: 45000, // optional, default: 45000
gzip: true, // optional, default: true
publicBasePath: '/sitemaps/', // optional, default: './'
xslUrl: 'https://example.com/sitemap.xsl', // optional XSL stylesheet
}).then(() => {
// Do follow up actions
})
```
Want to customize that?
```js
import { createReadStream, createWriteStream } from 'fs'
import { resolve } from 'path'
import { createGzip } from 'zlib'
import { Readable, pipeline } from 'stream'
import { SitemapAndIndexStream, SitemapStream, lineSeparatedURLsToSitemapOptions } from 'sitemap'
const sms = new SitemapAndIndexStream({
limit: 50000, // defaults to 45000
lastmodDateOnly: false, // print date not time
// SitemapAndIndexStream will call this user provided function every time
// it needs to create a new sitemap file. You merely need to return a stream
// for it to write the sitemap urls to and the expected url where that sitemap will be hosted
getSitemapStream: (i) => {
const sitemapStream = new SitemapStream({ hostname: 'https://example.com' });
// if your server automatically serves sitemap.xml.gz when requesting sitemap.xml leave this line be
// otherwise you will need to add .gz here and remove it a couple lines below so that both the index
// and the actual file have a .gz extension
const path = `./sitemap-${i}.xml`;
const ws = createWriteStream(resolve(path + '.gz')); // write it to sitemap-NUMBER.xml
// Use pipeline() rather than pipe() so a gzip failure also fails ws,
// which SitemapAndIndexStream watches. Passing the error on to sms also
// reports a failure that comes after sms stopped watching ws.
pipeline(sitemapStream, createGzip(), ws, (err) => { // compress the output of the sitemap
if (err) sms.destroy(err);
});
return [new URL(path, 'https://example.com/subdir/').toString(), sitemapStream, ws];
},
});
// when reading from a file
lineSeparatedURLsToSitemapOptions(
createReadStream('./your-data.json.txt')
)
.pipe(sms)
.pipe(createGzip())
.pipe(createWriteStream(resolve('./sitemap-index.xml.gz')));
// or reading straight from an in-memory array
sms
.pipe(createGzip())
.pipe(createWriteStream(resolve('./sitemap-index.xml.gz')));
const arrayOfSitemapItems = [{ url: '/page-1/', changefreq: 'daily'}, ...]
Readable.from(arrayOfSitemapItems).pipe(sms)
// or
arrayOfSitemapItems.forEach(item => sms.write(item))
sms.end() // necessary to let it know you've got nothing else to write
```
### Options you can pass
```js
import { SitemapStream, streamToPromise } from 'sitemap'
const smStream = new SitemapStream({
hostname: 'http://www.mywebsite.com',
xslUrl: "https://example.com/style.xsl",
lastmodDateOnly: false, // print date not time
xmlns: { // trim the xml namespace
news: true, // flip to false to omit the xml namespace for news
xhtml: true,
image: true,
video: true,
custom: [
'xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd"',
'xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"',
],
}
})
// coalesce stream to value
// alternatively you can pipe to another stream
streamToPromise(smStream).then(console.log)
smStream.write({
url: '/page1',
changefreq: 'weekly',
priority: 0.8, // A hint to the crawler that it should prioritize this over items less than 0.8
})
// each sitemap entry supports many options
// See [Sitemap Item Options](./api.md#sitemap-item-options) below for details
smStream.write({
url: 'http://test.com/page-1/',
img: [
{
url: 'http://test.com/img1.jpg',
caption: 'An image',
title: 'The Title of Image One',
geoLocation: 'London, United Kingdom',
license: 'https://creativecommons.org/licenses/by/4.0/'
},
{
url: 'http://test.com/img2.jpg',
caption: 'Another image',
title: 'The Title of Image Two',
geoLocation: 'London, United Kingdom',
license: 'https://creativecommons.org/licenses/by/4.0/'
}
],
video: [
{
thumbnail_loc: 'http://test.com/tmbn1.jpg',
title: 'A video title',
description: 'This is a video'
},
{
thumbnail_loc: 'http://test.com/tmbn2.jpg',
title: 'A video with an attribute',
description: 'This is another video',
'player_loc': 'http://www.example.com/videoplayer.mp4?video=123',
'player_loc:autoplay': 'ap=1',
'player_loc:allow_embed': 'yes'
}
],
links: [
{ lang: 'en', url: 'http://test.com/page-1/' },
{ lang: 'ja', url: 'http://test.com/page-1/ja/' }
],
androidLink: 'android-app://com.company.test/page-1/',
news: {
publication: {
name: 'The Example Times',
language: 'en'
},
genres: 'PressRelease, Blog',
publication_date: '2008-12-23',
title: 'Companies A, B in Merger Talks',
keywords: 'business, merger, acquisition, A, B',
stock_tickers: 'NASDAQ:A, NASDAQ:B'
}
})
// indicate there is nothing left to write
smStream.end()
```
## Filtering sitemap entries during parsing
You can filter or delete items from a sitemap while parsing by piping through a custom Transform stream. This is useful when you want to selectively process only certain URLs from an existing sitemap.
```js
import { createReadStream } from 'fs'
import { Transform } from 'stream'
import { XMLToSitemapItemStream } from 'sitemap'
// Create a filter that only keeps certain URLs
const filterStream = new Transform({
objectMode: true,
transform(item, encoding, callback) {
// Only keep URLs containing '/blog/'
if (item.url.includes('/blog/')) {
callback(undefined, item) // Keep this item
} else {
callback() // Skip this item (effectively "deleting" it)
}
}
})
// Parse and filter
createReadStream('./sitemap.xml')
.pipe(new XMLToSitemapItemStream())
.pipe(filterStream)
.on('data', (item) => {
console.log('Filtered URL:', item.url)
})
```
You can also chain multiple filters together, filter based on priority/changefreq, or use the filtered results to generate a new sitemap. See [examples/filter-sitemap.js](./examples/filter-sitemap.js) for more filtering patterns.
## Examples
For more examples see the [examples directory](./examples/)
## API
Full API docs can be found [here](./api.md)
## Maintainers
- [@ekalinin](https://github.com/ekalinin)
- [@derduher](https://github.com/derduher)
## License
See [LICENSE](https://github.com/ekalinin/sitemap.js/blob/master/LICENSE) file.
+2
View File
@@ -0,0 +1,2 @@
#!/usr/bin/env node
export {};
Generated Vendored Executable
+238
View File
@@ -0,0 +1,238 @@
#!/usr/bin/env node
import { createReadStream, readFileSync } from 'node:fs';
import { Readable, pipeline } from 'node:stream';
import { URL } from 'node:url';
import { createGzip } from 'node:zlib';
import { parseArgs } from 'node:util';
import { xmlLint } from './lib/xmllint.js';
import { InvalidLimitError, XMLLintUnavailable } from './lib/errors.js';
import { ObjectStreamToJSON, XMLToSitemapItemStream, } from './lib/sitemap-parser.js';
import { findUp, lineSeparatedURLsToSitemapOptions, openSitemapFile, } from './lib/utils.js';
import { SitemapStream } from './lib/sitemap-stream.js';
import { SitemapAndIndexStream } from './lib/sitemap-index-stream.js';
import { ErrorLevel } from './lib/types.js';
import { validateLimit } from './lib/validation.js';
/**
* Locate the package manifest by walking up from this module's directory.
*
* Resolving it at a hardcoded relative depth has broken twice already, because
* the correct depth depends on the build's output layout (`dist/esm/cli.js`
* needed `../../`, `dist/cli.js` needs `../`). Walking up finds it whichever
* layout this file is running from, source or built.
*/
const findPackageJson = () => {
const packageJson = findUp('package.json', import.meta.dirname);
if (!packageJson) {
throw new Error('Could not locate package.json');
}
return packageJson;
};
const packageJson = JSON.parse(readFileSync(findPackageJson(), 'utf8'));
const parseCliArgs = () => {
try {
return parseArgs({
allowPositionals: true,
options: {
help: { type: 'boolean', short: 'h' },
version: { type: 'boolean' },
validate: { type: 'boolean' },
index: { type: 'boolean' },
'index-base-url': { type: 'string' },
limit: { type: 'string' },
parse: { type: 'boolean' },
'single-line-json': { type: 'boolean' },
prepend: { type: 'string' },
gzip: { type: 'boolean' },
},
});
}
catch (err) {
console.error(err instanceof Error ? err.message : String(err));
console.error('Run `sitemap --help` for a list of options.');
process.exit(2);
}
};
const { values: argv, positionals } = parseCliArgs();
const pickStreamOrArg = () => {
if (!positionals.length) {
return process.stdin;
}
else {
return createReadStream(positionals[0], { encoding: 'utf8' });
}
};
/**
* Pipe `streams` together and write the result to stdout. On failure, print
* the error message (not an unhandled-error stack trace) and exit with 1.
*
* With `quietEPIPE`, a reader that closes stdout early (such as `| head`) is
* not a failure. Leave it off when the pipeline writes files as it goes, since
* stopping early leaves them incomplete.
*/
function pipeToStdout(streams, { quietEPIPE }) {
pipeline([...streams, process.stdout], (err) => {
if (!err ||
(quietEPIPE && err.code === 'EPIPE')) {
return;
}
console.error(err.message);
process.exitCode = 1;
});
}
/**
* Yield every item of each source in turn, so later sources start only after
* earlier ones have ended
*/
async function* concatSources(...sources) {
for (const source of sources) {
yield* source();
}
}
function getStream() {
if (positionals.length) {
return createReadStream(positionals[0]);
}
else {
console.warn('Reading from stdin. If you are not piping anything in, this command is not doing anything');
return process.stdin;
}
}
if (argv.version) {
console.log(packageJson.version);
}
else if (argv.help) {
console.log(`
Turn a list of urls into a sitemap xml.
Options:
--help Print this text
--version Print the version
--validate Ensure the passed in file is conforms to the sitemap spec
--index Create an index and stream that out. Writes out sitemaps along the way.
--index-base-url Base url the sitemaps will be hosted eg. https://example.com/sitemaps/
--limit=45000 Set a custom limit to the items per sitemap
--parse Parse fed xml and spit out config
--prepend=sitemap.xml Prepend the streamed in sitemap configs to sitemap.xml
--gzip Compress output. With --index, also gzips each sitemap
and names it sitemap-N.xml.gz
--single-line-json When used with parse, it spits out each entry as json rather than the whole json.
# examples
Generate a sitemap index file as well as sitemaps
npx sitemap --gzip --index --index-base-url https://example.com/path/to/sitemaps/ < listofurls.txt > sitemap-index.xml.gz
Add to a sitemap
npx sitemap --prepend sitemap.xml < listofurls.json
Turn an existing sitemap into configuration understood by the sitemap library
npx sitemap --parse sitemap.xml
Use XMLLib to validate your sitemap (requires xmllib)
npx sitemap --validate sitemap.xml
`);
}
else if (argv.parse) {
const streams = [
getStream(),
new XMLToSitemapItemStream({ level: ErrorLevel.THROW }),
new ObjectStreamToJSON({ lineSeparated: !argv['single-line-json'] }),
];
if (argv.gzip) {
streams.push(createGzip());
}
pipeToStdout(streams, { quietEPIPE: true });
}
else if (argv.validate) {
xmlLint(getStream())
.then(() => console.log('valid'))
.catch(([error, stderr]) => {
if (error instanceof XMLLintUnavailable || stderr === undefined) {
// No xmllint output to show (xmllint missing, or the schema could
// not be found), so report the error itself. Exit 2: validation
// could not run, as distinct from a file that failed it.
console.error(error?.message);
process.exitCode = 2;
return;
}
else {
console.log(stderr);
// Exit 1 so scripts and CI can detect an invalid sitemap
process.exitCode = 1;
}
});
}
else if (argv.index) {
// Decimal digits only: Number() alone would also accept forms such as
// "1e3", "0x10" or " 250 ". Anything else becomes NaN, which fails below.
const limit = argv.limit === undefined
? undefined
: /^\d+$/.test(argv.limit)
? Number(argv.limit)
: NaN;
try {
if (limit !== undefined) {
validateLimit(limit);
}
}
catch {
// Report the value as typed, rather than the NaN it parsed to
console.error(`--limit: ${new InvalidLimitError(argv.limit).message}`);
process.exit(2);
}
const baseURL = argv['index-base-url'];
if (!baseURL) {
throw new Error("You must specify where the sitemaps will be hosted. use --index-base-url 'https://example.com/path'");
}
const sms = new SitemapAndIndexStream({
limit,
getSitemapStream: (i) =>
// Gzipped files are named .xml.gz, as simpleSitemapAndIndex names them
openSitemapFile(i, new SitemapStream(), {
destinationDir: '.',
gzip: Boolean(argv.gzip),
toURL: (fileName) => new URL(fileName, baseURL).toString(),
urlParamName: '--index-base-url',
onError: (err) => sms.destroy(err),
}),
});
const streams = [
lineSeparatedURLsToSitemapOptions(pickStreamOrArg()),
sms,
];
if (argv.gzip) {
streams.push(createGzip());
}
// sitemap files are written as the index streams out, so a reader that
// stops early leaves them incomplete: report it
pipeToStdout(streams, { quietEPIPE: false });
}
else {
const sources = [];
const prepend = argv.prepend;
if (prepend) {
// Read the existing sitemap to the end before the new URLs, so its
// entries come first and one source ending can't end the sitemap early
sources.push(() => {
const parser = new XMLToSitemapItemStream();
// Past the cap the parser drops entries and only logs it, which would
// write a truncated sitemap and exit 0. Fail instead: the output can't
// hold more than a sitemap may.
parser.once('maxEntriesExceeded', (max) => {
parser.destroy(new Error(`--prepend file ${prepend} has more than ${max} URLs, the most a sitemap may hold`));
});
// pipeline() destroys both streams if either fails, and fails the
// parser with the error, which concatSources then rethrows. The
// callback has nothing to add.
return pipeline(createReadStream(prepend), parser, () => { });
});
}
sources.push(() => lineSeparatedURLsToSitemapOptions(pickStreamOrArg()));
const streams = [
Readable.from(concatSources(...sources)),
new SitemapStream(),
];
if (argv.gzip) {
streams.push(createGzip());
}
pipeToStdout(streams, { quietEPIPE: true });
}
+17
View File
@@ -0,0 +1,17 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
export { SitemapItemStream, type SitemapItemStreamOptions, } from './lib/sitemap-item-stream.js';
export { IndexTagNames, SitemapIndexStream, type SitemapIndexStreamOptions, SitemapAndIndexStream, type SitemapAndIndexStreamOptions, } from './lib/sitemap-index-stream.js';
export { streamToPromise, SitemapStream, type SitemapStreamOptions, } from './lib/sitemap-stream.js';
export { NoURLError, NoConfigError, ChangeFreqInvalidError, PriorityInvalidError, InvalidVideoFormat, InvalidVideoDuration, InvalidVideoDescription, InvalidVideoRating, InvalidAttrValue, InvalidAttr, InvalidNewsFormat, InvalidNewsAccessValue, XMLLintUnavailable, InvalidVideoTitle, InvalidVideoViewCount, InvalidVideoTagCount, InvalidVideoCategory, InvalidVideoFamilyFriendly, InvalidVideoYesNo, InvalidLinkHreflang, InvalidVideoRestriction, InvalidVideoRestrictionRelationship, InvalidVideoPriceType, InvalidVideoResolution, InvalidVideoPriceCurrency, EmptyStream, EmptySitemap, InvalidPathError, InvalidHostnameError, InvalidURLError, InvalidLimitError, InvalidPublicBasePathError, InvalidXSLUrlError, InvalidXMLAttributeNameError, } from './lib/errors.js';
export { EnumChangefreq, EnumYesNo, EnumAllowDeny, ErrorLevel, TagNames, type NewsItem, type Img, type PriceType, type Resolution, type VideoItem, type VideoItemLoose, type LinkItem, type IndexItem, type SitemapItem, type SitemapItemLoose, type ErrorHandler, type Logger, type StringObj, type SaxParserStreamOptions, } from './lib/types.js';
export { lineSeparatedURLsToSitemapOptions, mergeStreams, normalizeURL, ReadlineStream, type ReadlineStreamOptions, } from './lib/utils.js';
export { xmlLint } from './lib/xmllint.js';
export { parseSitemap, XMLToSitemapItemStream, type XMLToSitemapItemStreamOptions, ObjectStreamToJSON, type ObjectStreamToJSONOptions, } from './lib/sitemap-parser.js';
export { parseSitemapIndex, XMLToSitemapIndexStream, type XMLToSitemapIndexItemStreamOptions, IndexObjectStreamToJSON, type IndexObjectStreamToJSONOptions, } from './lib/sitemap-index-parser.js';
export { simpleSitemapAndIndex, type SimpleSitemapAndIndexOptions, } from './lib/sitemap-simple.js';
export { validateSMIOptions, validateURL, validatePath, validateLimit, validatePublicBasePath, validateXSLUrl, validators, isPriceType, isResolution, isValidChangeFreq, isValidYesNo, isAllowDeny, } from './lib/validation.js';
export { LIMITS, DEFAULT_SITEMAP_ITEM_LIMIT } from './lib/constants.js';
+22
View File
@@ -0,0 +1,22 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
export { SitemapItemStream, } from './lib/sitemap-item-stream.js';
export { IndexTagNames, SitemapIndexStream, SitemapAndIndexStream, } from './lib/sitemap-index-stream.js';
export { streamToPromise, SitemapStream, } from './lib/sitemap-stream.js';
// Listed explicitly, like the types below, so a new error class is public
// only once it is added here
export { NoURLError, NoConfigError, ChangeFreqInvalidError, PriorityInvalidError, InvalidVideoFormat, InvalidVideoDuration, InvalidVideoDescription, InvalidVideoRating, InvalidAttrValue, InvalidAttr, InvalidNewsFormat, InvalidNewsAccessValue, XMLLintUnavailable, InvalidVideoTitle, InvalidVideoViewCount, InvalidVideoTagCount, InvalidVideoCategory, InvalidVideoFamilyFriendly, InvalidVideoYesNo, InvalidLinkHreflang, InvalidVideoRestriction, InvalidVideoRestrictionRelationship, InvalidVideoPriceType, InvalidVideoResolution, InvalidVideoPriceCurrency, EmptyStream, EmptySitemap, InvalidPathError, InvalidHostnameError, InvalidURLError, InvalidLimitError, InvalidPublicBasePathError, InvalidXSLUrlError, InvalidXMLAttributeNameError, } from './lib/errors.js';
// Listed explicitly rather than `export *`, so types.ts can hold internal types
// (such as SitemapFileOptions) without making them public API. IndexTagNames
// and the parser options are exported with their modules below.
export { EnumChangefreq, EnumYesNo, EnumAllowDeny, ErrorLevel, TagNames, } from './lib/types.js';
export { lineSeparatedURLsToSitemapOptions, mergeStreams, normalizeURL, ReadlineStream, } from './lib/utils.js';
export { xmlLint } from './lib/xmllint.js';
export { parseSitemap, XMLToSitemapItemStream, ObjectStreamToJSON, } from './lib/sitemap-parser.js';
export { parseSitemapIndex, XMLToSitemapIndexStream, IndexObjectStreamToJSON, } from './lib/sitemap-index-parser.js';
export { simpleSitemapAndIndex, } from './lib/sitemap-simple.js';
export { validateSMIOptions, validateURL, validatePath, validateLimit, validatePublicBasePath, validateXSLUrl, validators, isPriceType, isResolution, isValidChangeFreq, isValidYesNo, isAllowDeny, } from './lib/validation.js';
export { LIMITS, DEFAULT_SITEMAP_ITEM_LIMIT } from './lib/constants.js';
+65
View File
@@ -0,0 +1,65 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
/**
* Shared constants used across the sitemap library
* This file serves as a single source of truth for limits and validation patterns
*/
/**
* Security limits for sitemap generation and parsing
*
* These limits are based on:
* - sitemaps.org protocol specification
* - Security best practices to prevent DoS and injection attacks
* - Google's sitemap extension specifications
*
* @see https://www.sitemaps.org/protocol.html
* @see https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap
*/
export declare const LIMITS: {
readonly MAX_URL_LENGTH: 2048;
readonly URL_PROTOCOL_REGEX: RegExp;
readonly MIN_SITEMAP_ITEM_LIMIT: 1;
readonly MAX_SITEMAP_ITEM_LIMIT: 50000;
readonly MIN_PRIORITY: 0;
readonly MAX_PRIORITY: 1;
readonly PRIORITY_STRING_REGEX: RegExp;
readonly MIN_VIDEO_DURATION: 0;
readonly MAX_VIDEO_DURATION: 28800;
readonly MIN_VIDEO_RATING: 0;
readonly MAX_VIDEO_RATING: 5;
readonly MIN_VIDEO_VIEW_COUNT: 0;
readonly MAX_VIDEO_TITLE_LENGTH: 100;
readonly MAX_VIDEO_DESCRIPTION_LENGTH: 2048;
readonly MAX_VIDEO_CATEGORY_LENGTH: 256;
readonly MAX_TAGS_PER_VIDEO: 32;
readonly MAX_NEWS_TITLE_LENGTH: 200;
readonly MAX_NEWS_NAME_LENGTH: 256;
readonly MAX_IMAGE_CAPTION_LENGTH: 512;
readonly MAX_IMAGE_TITLE_LENGTH: 512;
readonly MAX_IMAGES_PER_URL: 1000;
readonly MAX_VIDEOS_PER_URL: 100;
readonly MAX_LINKS_PER_URL: 100;
/**
* Total entries in a sitemap.
* @deprecated Unused; the parsers cap entries at MAX_SITEMAP_ITEM_LIMIT
* (same value), overridable with their `maxEntries` option.
*/
readonly MAX_URL_ENTRIES: 50000;
readonly ISO_DATE_REGEX: RegExp;
readonly MAX_CUSTOM_NAMESPACES: 20;
readonly MAX_NAMESPACE_LENGTH: 512;
readonly MAX_PARSER_CHAR_DATA_LENGTH: 65536;
readonly MAX_PARSER_ERRORS: 100;
};
/**
* Default maximum number of items in each sitemap XML file, shared by
* SitemapAndIndexStream, simpleSitemapAndIndex and the CLI.
* Set below the 50,000 max to leave headroom under the 50MB uncompressed
* file-size limit, which entries with images, video or news can approach.
*
* @see https://www.sitemaps.org/protocol.html#index
*/
export declare const DEFAULT_SITEMAP_ITEM_LIMIT = 45000;
+81
View File
@@ -0,0 +1,81 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
/**
* Shared constants used across the sitemap library
* This file serves as a single source of truth for limits and validation patterns
*/
/**
* Security limits for sitemap generation and parsing
*
* These limits are based on:
* - sitemaps.org protocol specification
* - Security best practices to prevent DoS and injection attacks
* - Google's sitemap extension specifications
*
* @see https://www.sitemaps.org/protocol.html
* @see https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap
*/
export const LIMITS = {
// URL constraints per sitemaps.org spec
MAX_URL_LENGTH: 2048,
URL_PROTOCOL_REGEX: /^https?:\/\//i,
// Sitemap size limits per sitemaps.org spec
MIN_SITEMAP_ITEM_LIMIT: 1,
MAX_SITEMAP_ITEM_LIMIT: 50000,
// Priority bounds per sitemaps.org spec
MIN_PRIORITY: 0,
MAX_PRIORITY: 1,
// A priority string normalizeURL converts to a number: plain decimal only,
// so hex, binary and exponent forms that Number() accepts stay invalid
PRIORITY_STRING_REGEX: /^\s*(?:\d+\.?\d*|\.\d+)\s*$/,
// Video value bounds per Google spec
MIN_VIDEO_DURATION: 0,
MAX_VIDEO_DURATION: 28800, // 8 hours
MIN_VIDEO_RATING: 0,
MAX_VIDEO_RATING: 5,
MIN_VIDEO_VIEW_COUNT: 0,
// Video field length constraints per Google spec
MAX_VIDEO_TITLE_LENGTH: 100,
MAX_VIDEO_DESCRIPTION_LENGTH: 2048,
MAX_VIDEO_CATEGORY_LENGTH: 256,
MAX_TAGS_PER_VIDEO: 32,
// News field length constraints per Google spec
MAX_NEWS_TITLE_LENGTH: 200,
MAX_NEWS_NAME_LENGTH: 256,
// Image field length constraints per Google spec
MAX_IMAGE_CAPTION_LENGTH: 512,
MAX_IMAGE_TITLE_LENGTH: 512,
// Limits on number of items per URL entry
MAX_IMAGES_PER_URL: 1000,
MAX_VIDEOS_PER_URL: 100,
MAX_LINKS_PER_URL: 100,
/**
* Total entries in a sitemap.
* @deprecated Unused; the parsers cap entries at MAX_SITEMAP_ITEM_LIMIT
* (same value), overridable with their `maxEntries` option.
*/
MAX_URL_ENTRIES: 50000,
// Date validation - ISO 8601 / W3C format
ISO_DATE_REGEX: /^\d{4}-\d{2}-\d{2}(T\d{2}:\d{2}:\d{2}(\.\d{3})?([+-]\d{2}:\d{2}|Z)?)?$/,
// Custom namespace limits to prevent DoS
MAX_CUSTOM_NAMESPACES: 20,
MAX_NAMESPACE_LENGTH: 512,
// Cap on buffered character data per element in the parser, matching sax's
// default buffer size. Larger values are dropped to bound memory use.
MAX_PARSER_CHAR_DATA_LENGTH: 65536,
// Cap on stored parser errors to prevent memory DoS (BB-03)
// Errors beyond this limit are counted in errorCount but not retained as objects
MAX_PARSER_ERRORS: 100,
};
/**
* Default maximum number of items in each sitemap XML file, shared by
* SitemapAndIndexStream, simpleSitemapAndIndex and the CLI.
* Set below the 50,000 max to leave headroom under the 50MB uncompressed
* file-size limit, which entries with images, video or news can approach.
*
* @see https://www.sitemaps.org/protocol.html#index
*/
export const DEFAULT_SITEMAP_ITEM_LIMIT = 45000;
+131
View File
@@ -0,0 +1,131 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
/**
* URL in SitemapItem does not exist
*/
export declare class NoURLError extends Error {
constructor(message?: string);
}
/**
* Config was not passed to SitemapItem constructor
*/
export declare class NoConfigError extends Error {
constructor(message?: string);
}
/**
* changefreq property in sitemap is invalid
*/
export declare class ChangeFreqInvalidError extends Error {
constructor(url: string, changefreq: any);
}
/**
* priority property in sitemap is invalid
*/
export declare class PriorityInvalidError extends Error {
constructor(url: string, priority: any);
}
export declare class InvalidVideoFormat extends Error {
constructor(url: string);
}
export declare class InvalidVideoDuration extends Error {
constructor(url: string, duration: any);
}
export declare class InvalidVideoDescription extends Error {
constructor(url: string, length: number);
}
export declare class InvalidVideoRating extends Error {
constructor(url: string, title: any, rating: any);
}
export declare class InvalidAttrValue extends Error {
constructor(key: string, val: any, validator: RegExp, url?: string);
}
export declare class InvalidAttr extends Error {
constructor(key: string);
}
export declare class InvalidNewsFormat extends Error {
constructor(url: string);
}
export declare class InvalidNewsAccessValue extends Error {
constructor(url: string, access: any);
}
export declare class XMLLintUnavailable extends Error {
constructor(message?: string);
}
export declare class InvalidVideoTitle extends Error {
constructor(url: string, length: number);
}
export declare class InvalidVideoViewCount extends Error {
constructor(url: string, count: number);
}
export declare class InvalidVideoTagCount extends Error {
constructor(url: string, count: number);
}
export declare class InvalidVideoCategory extends Error {
constructor(url: string, count: number);
}
export declare class InvalidVideoFamilyFriendly extends Error {
constructor(url: string, fam: string);
}
export declare class InvalidVideoYesNo extends Error {
constructor(url: string, field: string, value: string);
}
/**
* An alternate link has no `lang` or `hreflang`
*/
export declare class InvalidLinkHreflang extends Error {
constructor(url: string, href: string, value: unknown);
}
export declare class InvalidVideoRestriction extends Error {
constructor(url: string, code: string);
}
export declare class InvalidVideoRestrictionRelationship extends Error {
constructor(url: string, val?: string);
}
export declare class InvalidVideoPriceType extends Error {
constructor(url: string, priceType?: string, price?: string);
}
export declare class InvalidVideoResolution extends Error {
constructor(url: string, resolution: string);
}
export declare class InvalidVideoPriceCurrency extends Error {
constructor(url: string, currency: string);
}
export declare class EmptyStream extends Error {
constructor();
}
/**
* @deprecated No longer thrown: since 10.0.0, `SitemapStream` emits an empty
* `<urlset>` when it ends without items. Kept so existing imports still resolve.
*/
export declare class EmptySitemap extends Error {
constructor();
}
export declare class InvalidPathError extends Error {
constructor(path: string, reason: string);
}
export declare class InvalidHostnameError extends Error {
constructor(hostname: string, reason: string);
}
/**
* A URL failed validateURL. Extends InvalidHostnameError so existing
* `instanceof InvalidHostnameError` checks keep working; the offending value
* is truncated so attacker-sized input is not copied into every message.
*/
export declare class InvalidURLError extends InvalidHostnameError {
constructor(url: string, reason: string);
}
export declare class InvalidLimitError extends Error {
constructor(limit: any);
}
export declare class InvalidPublicBasePathError extends Error {
constructor(publicBasePath: string, reason: string);
}
export declare class InvalidXSLUrlError extends Error {
constructor(xslUrl: string, reason: string);
}
export declare class InvalidXMLAttributeNameError extends Error {
constructor(attributeName: string);
}
+284
View File
@@ -0,0 +1,284 @@
/* eslint-disable @typescript-eslint/no-explicit-any */
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
/**
* URL in SitemapItem does not exist
*/
export class NoURLError extends Error {
constructor(message) {
super(message || 'URL is required');
this.name = 'NoURLError';
Error.captureStackTrace(this, NoURLError);
}
}
/**
* Config was not passed to SitemapItem constructor
*/
export class NoConfigError extends Error {
constructor(message) {
super(message || 'SitemapItem requires a configuration');
this.name = 'NoConfigError';
Error.captureStackTrace(this, NoConfigError);
}
}
/**
* changefreq property in sitemap is invalid
*/
export class ChangeFreqInvalidError extends Error {
constructor(url, changefreq) {
super(`${url}: changefreq "${changefreq}" is invalid`);
this.name = 'ChangeFreqInvalidError';
Error.captureStackTrace(this, ChangeFreqInvalidError);
}
}
/**
* priority property in sitemap is invalid
*/
export class PriorityInvalidError extends Error {
constructor(url, priority) {
super(`${url}: priority "${priority}" must be a number between 0 and 1 inclusive`);
this.name = 'PriorityInvalidError';
Error.captureStackTrace(this, PriorityInvalidError);
}
}
export class InvalidVideoFormat extends Error {
constructor(url) {
super(`${url} video must include thumbnail_loc, title and description fields for videos`);
this.name = 'InvalidVideoFormat';
Error.captureStackTrace(this, InvalidVideoFormat);
}
}
export class InvalidVideoDuration extends Error {
constructor(url, duration) {
super(`${url} duration "${duration}" must be an integer of seconds between 0 and 28800`);
this.name = 'InvalidVideoDuration';
Error.captureStackTrace(this, InvalidVideoDuration);
}
}
export class InvalidVideoDescription extends Error {
constructor(url, length) {
const message = `${url}: video description is too long ${length} vs limit of 2048 characters.`;
super(message);
this.name = 'InvalidVideoDescription';
Error.captureStackTrace(this, InvalidVideoDescription);
}
}
export class InvalidVideoRating extends Error {
constructor(url, title, rating) {
super(`${url}: video "${title}" rating "${rating}" must be between 0 and 5 inclusive`);
this.name = 'InvalidVideoRating';
Error.captureStackTrace(this, InvalidVideoRating);
}
}
export class InvalidAttrValue extends Error {
constructor(key, val, validator, url) {
super((url ? `${url}: ` : '') +
'"' +
val +
'" tested against: ' +
validator +
' is not a valid value for attr: "' +
key +
'"');
this.name = 'InvalidAttrValue';
Error.captureStackTrace(this, InvalidAttrValue);
}
}
// InvalidAttr is only thrown when attrbuilder is called incorrectly internally
/* istanbul ignore next */
export class InvalidAttr extends Error {
constructor(key) {
super('"' + key + '" is malformed');
this.name = 'InvalidAttr';
Error.captureStackTrace(this, InvalidAttr);
}
}
export class InvalidNewsFormat extends Error {
constructor(url) {
super(`${url} News must include publication, publication name, publication language, title, and publication_date for news`);
this.name = 'InvalidNewsFormat';
Error.captureStackTrace(this, InvalidNewsFormat);
}
}
export class InvalidNewsAccessValue extends Error {
constructor(url, access) {
super(`${url} News access "${access}" must be either Registration, Subscription or not be present`);
this.name = 'InvalidNewsAccessValue';
Error.captureStackTrace(this, InvalidNewsAccessValue);
}
}
export class XMLLintUnavailable extends Error {
constructor(message) {
super(message || 'xmlLint is not installed. XMLLint is required to validate');
this.name = 'XMLLintUnavailable';
Error.captureStackTrace(this, XMLLintUnavailable);
}
}
export class InvalidVideoTitle extends Error {
constructor(url, length) {
super(`${url}: video title is too long ${length} vs 100 character limit`);
this.name = 'InvalidVideoTitle';
Error.captureStackTrace(this, InvalidVideoTitle);
}
}
export class InvalidVideoViewCount extends Error {
constructor(url, count) {
super(`${url}: video view count must be positive, view count was ${count}`);
this.name = 'InvalidVideoViewCount';
Error.captureStackTrace(this, InvalidVideoViewCount);
}
}
export class InvalidVideoTagCount extends Error {
constructor(url, count) {
super(`${url}: video can have no more than 32 tags, this has ${count}`);
this.name = 'InvalidVideoTagCount';
Error.captureStackTrace(this, InvalidVideoTagCount);
}
}
export class InvalidVideoCategory extends Error {
constructor(url, count) {
super(`${url}: video category can only be 256 characters but was passed ${count}`);
this.name = 'InvalidVideoCategory';
Error.captureStackTrace(this, InvalidVideoCategory);
}
}
export class InvalidVideoFamilyFriendly extends Error {
constructor(url, fam) {
super(`${url}: video family friendly must be yes or no, was passed "${fam}"`);
this.name = 'InvalidVideoFamilyFriendly';
Error.captureStackTrace(this, InvalidVideoFamilyFriendly);
}
}
export class InvalidVideoYesNo extends Error {
constructor(url, field, value) {
super(`${url}: video ${field} must be yes or no, was passed "${value}"`);
this.name = 'InvalidVideoYesNo';
Error.captureStackTrace(this, InvalidVideoYesNo);
}
}
/**
* An alternate link has no `lang` or `hreflang`
*/
export class InvalidLinkHreflang extends Error {
constructor(url, href, value) {
super(`${url}: alternate link to "${href}" needs a lang or hreflang string, was passed "${String(value)}"`);
this.name = 'InvalidLinkHreflang';
Error.captureStackTrace(this, InvalidLinkHreflang);
}
}
export class InvalidVideoRestriction extends Error {
constructor(url, code) {
super(`${url}: video restriction must be one or more two letter country codes. Was passed "${code}"`);
this.name = 'InvalidVideoRestriction';
Error.captureStackTrace(this, InvalidVideoRestriction);
}
}
export class InvalidVideoRestrictionRelationship extends Error {
constructor(url, val) {
super(`${url}: video restriction relationship must be either allow or deny. Was passed "${val}"`);
this.name = 'InvalidVideoRestrictionRelationship';
Error.captureStackTrace(this, InvalidVideoRestrictionRelationship);
}
}
export class InvalidVideoPriceType extends Error {
constructor(url, priceType, price) {
super(priceType === undefined && price === ''
? `${url}: video priceType is required when price is not provided`
: `${url}: video price type "${priceType}" is not "rent" or "purchase"`);
this.name = 'InvalidVideoPriceType';
Error.captureStackTrace(this, InvalidVideoPriceType);
}
}
export class InvalidVideoResolution extends Error {
constructor(url, resolution) {
super(`${url}: video price resolution "${resolution}" is not hd or sd`);
this.name = 'InvalidVideoResolution';
Error.captureStackTrace(this, InvalidVideoResolution);
}
}
export class InvalidVideoPriceCurrency extends Error {
constructor(url, currency) {
super(`${url}: video price currency "${currency}" must be a three capital letter abbrieviation for the country currency`);
this.name = 'InvalidVideoPriceCurrency';
Error.captureStackTrace(this, InvalidVideoPriceCurrency);
}
}
export class EmptyStream extends Error {
constructor() {
super('You have ended the stream before anything was written. streamToPromise MUST be called before ending the stream.');
this.name = 'EmptyStream';
Error.captureStackTrace(this, EmptyStream);
}
}
/**
* @deprecated No longer thrown: since 10.0.0, `SitemapStream` emits an empty
* `<urlset>` when it ends without items. Kept so existing imports still resolve.
*/
export class EmptySitemap extends Error {
constructor() {
super('You ended the stream without writing anything.');
this.name = 'EmptySitemap';
Error.captureStackTrace(this, EmptySitemap);
}
}
export class InvalidPathError extends Error {
constructor(path, reason) {
super(`Invalid path "${path}": ${reason}`);
this.name = 'InvalidPathError';
Error.captureStackTrace(this, InvalidPathError);
}
}
export class InvalidHostnameError extends Error {
constructor(hostname, reason) {
super(`Invalid hostname "${hostname}": ${reason}`);
this.name = 'InvalidHostnameError';
Error.captureStackTrace(this, InvalidHostnameError);
}
}
/**
* A URL failed validateURL. Extends InvalidHostnameError so existing
* `instanceof InvalidHostnameError` checks keep working; the offending value
* is truncated so attacker-sized input is not copied into every message.
*/
export class InvalidURLError extends InvalidHostnameError {
constructor(url, reason) {
super(url, reason);
const shown = typeof url === 'string' && url.length > 100
? `${url.substring(0, 100)}...`
: url;
this.message = `Invalid URL "${shown}": ${reason}`;
this.name = 'InvalidURLError';
Error.captureStackTrace(this, InvalidURLError);
}
}
export class InvalidLimitError extends Error {
constructor(limit) {
super(`Invalid limit "${limit}": must be an integer between 1 and 50000 (per sitemaps.org spec)`);
this.name = 'InvalidLimitError';
Error.captureStackTrace(this, InvalidLimitError);
}
}
export class InvalidPublicBasePathError extends Error {
constructor(publicBasePath, reason) {
super(`Invalid publicBasePath "${publicBasePath}": ${reason}`);
this.name = 'InvalidPublicBasePathError';
Error.captureStackTrace(this, InvalidPublicBasePathError);
}
}
export class InvalidXSLUrlError extends Error {
constructor(xslUrl, reason) {
super(`Invalid xslUrl "${xslUrl}": ${reason}`);
this.name = 'InvalidXSLUrlError';
Error.captureStackTrace(this, InvalidXSLUrlError);
}
}
export class InvalidXMLAttributeNameError extends Error {
constructor(attributeName) {
super(`Invalid XML attribute name "${attributeName}": must contain only alphanumeric characters, hyphens, underscores, and colons`);
this.name = 'InvalidXMLAttributeNameError';
Error.captureStackTrace(this, InvalidXMLAttributeNameError);
}
}
+21
View File
@@ -0,0 +1,21 @@
/**
* The logger helpers every stream shares. Kept apart from utils.ts so
* validation.ts can use them without importing utils.ts, which imports
* validation.ts.
*/
import type { ErrorHandler, Logger } from './types.js';
import { ErrorLevel } from './types.js';
/**
* The logger streams use when none is given: forwards to the console
*/
export declare const defaultLogger: Logger;
/**
* An ErrorHandler for validateSMIOptions that throws at ErrorLevel.THROW and
* sends warnings to the given logger at ErrorLevel.WARN
*/
export declare function loggingErrorHandler(logger: Logger): ErrorHandler;
/**
* Pick the logger a stream should use: a no-op when the level is SILENT or the
* caller passed `false`, otherwise the caller's logger or the console.
*/
export declare function resolveLogger(level: ErrorLevel, logger: Logger | false | undefined): Logger;
+30
View File
@@ -0,0 +1,30 @@
import { ErrorLevel } from './types.js';
/**
* The logger streams use when none is given: forwards to the console
*/
export const defaultLogger = (level, ...message) => console[level](...message);
const noopLogger = () => undefined;
/**
* An ErrorHandler for validateSMIOptions that throws at ErrorLevel.THROW and
* sends warnings to the given logger at ErrorLevel.WARN
*/
export function loggingErrorHandler(logger) {
return (error, level) => {
if (level === ErrorLevel.THROW) {
throw error;
}
else if (level === ErrorLevel.WARN) {
logger('warn', error.name, error.message);
}
};
}
/**
* Pick the logger a stream should use: a no-op when the level is SILENT or the
* caller passed `false`, otherwise the caller's logger or the console.
*/
export function resolveLogger(level, logger) {
if (level === ErrorLevel.SILENT || logger === false) {
return noopLogger;
}
return logger ?? defaultLogger;
}
+63
View File
@@ -0,0 +1,63 @@
import type { QualifiedAttribute, QualifiedTag, SAXStream } from 'sax';
import { type Readable, Transform, type TransformCallback } from 'node:stream';
import { type Logger, type SaxParserStreamConfig, type SaxParserStreamOptions, ErrorLevel } from './types.js';
/**
* Shared plumbing for the XML parser streams: sax setup, character-data
* buffering, error collection, backpressure, and the entry cap. Subclasses
* supply tag semantics through the `on*` hooks and hand each finished
* entry to `emitEntry`.
*
* Emits `'maxEntriesExceeded'` (with the cap) once, when the first entry past
* `maxEntries` is seen. If nothing listens for that event, the stream logs the
* overflow through its logger instead.
*/
export declare abstract class SaxParserStream<T> extends Transform {
level: ErrorLevel;
logger: Logger;
/**
* Errors encountered during parsing, capped at LIMITS.MAX_PARSER_ERRORS entries
* to prevent memory DoS from malformed XML (BB-03).
* Use errorCount for the total number of errors regardless of the cap.
*/
errors: Error[];
/** Total number of errors seen, including those beyond the stored cap. */
errorCount: number;
/** Number of entries emitted or dropped past `maxEntries` */
entryCount: number;
saxStream: SAXStream;
private maxEntries;
private overflowMessage;
constructor(opts: SaxParserStreamOptions, config: SaxParserStreamConfig);
/** The first parse error, or null if there has been none */
get error(): Error | null;
/** Called for every opening tag, after unknown tags have been reported */
protected onOpenTag(tag: QualifiedTag): void;
/** Called once per element with its joined, trimmed character data */
protected abstract onCharData(tag: string, text: string): void;
/** Called for each attribute of the tag currently being opened */
protected abstract onAttribute(tag: string, attr: QualifiedAttribute): void;
/** Called for every closing tag, after its character data is handled */
protected abstract onCloseTag(tag: string): void;
/** Count a finished entry and push it, unless it is past the entry cap */
protected emitEntry(item: T): void;
_transform(data: string, encoding: string, callback: TransformCallback): void;
/**
* Tell sax the document has ended, so a truncated document (an unclosed
* element) fails the stream like any other malformed XML.
*/
_flush(callback: TransformCallback): void;
/** Log a message and record it as a parse error, with the same text */
protected report(level: Parameters<Logger>[0], msg: string): void;
/** Record a parse error, keeping at most LIMITS.MAX_PARSER_ERRORS */
protected err(msg: string): void;
}
/**
* Pipe `xml` through `parser` and resolve with every entry it emits. Rejects
* if either stream fails.
*
* Entries past the parser's cap are dropped, so once the cap is crossed this
* stops reading `xml` and destroys it instead of parsing the rest (BB-05).
* `onOverflow` then picks the outcome: return an Error to reject with it, or
* nothing to resolve with the entries emitted before the cap.
*/
export declare function collectEntries<T>(xml: Readable, parser: SaxParserStream<T>, onOverflow: (maxEntries: number) => Error | void): Promise<T[]>;
+212
View File
@@ -0,0 +1,212 @@
import sax from 'sax';
import { Transform } from 'node:stream';
import { ErrorLevel, } from './types.js';
import { isValidMaxEntries } from './validation.js';
import { LIMITS } from './constants.js';
import { createCharDataBuffer } from './utils.js';
import { resolveLogger } from './logger.js';
/**
* Shared plumbing for the XML parser streams: sax setup, character-data
* buffering, error collection, backpressure, and the entry cap. Subclasses
* supply tag semantics through the `on*` hooks and hand each finished
* entry to `emitEntry`.
*
* Emits `'maxEntriesExceeded'` (with the cap) once, when the first entry past
* `maxEntries` is seen. If nothing listens for that event, the stream logs the
* overflow through its logger instead.
*/
export class SaxParserStream extends Transform {
level;
logger;
/**
* Errors encountered during parsing, capped at LIMITS.MAX_PARSER_ERRORS entries
* to prevent memory DoS from malformed XML (BB-03).
* Use errorCount for the total number of errors regardless of the cap.
*/
errors;
/** Total number of errors seen, including those beyond the stored cap. */
errorCount;
/** Number of entries emitted or dropped past `maxEntries` */
entryCount;
saxStream;
maxEntries;
overflowMessage;
constructor(opts, config) {
super({ ...opts, objectMode: true });
this.errors = [];
this.errorCount = 0;
this.entryCount = 0;
this.overflowMessage = config.overflowMessage;
this.maxEntries = opts.maxEntries ?? LIMITS.MAX_SITEMAP_ITEM_LIMIT;
if (!isValidMaxEntries(this.maxEntries)) {
throw new Error(`maxEntries must be a non-negative integer or Infinity, got ${this.maxEntries}`);
}
this.level = opts.level || ErrorLevel.WARN;
this.logger = resolveLogger(this.level, opts.logger);
this.saxStream = sax.createStream(true, {
xmlns: true,
// @ts-expect-error - SAX types don't include strictEntities option
strictEntities: true,
// No sax trim: it trims each text chunk separately, dropping whitespace
// between text and CDATA. The char-data buffer trims the joined value.
trim: false,
});
// Per ADR 0001, CDATA sections are ordinary character data: both SAX
// events append to one buffer, which hands the joined value for an
// element to onCharData.
const charData = createCharDataBuffer({
onText: (tag, text) => this.onCharData(tag, text),
onOverflow: (msg) => {
this.report('warn', msg);
},
});
let currentTag = '';
this.saxStream.on('opentagstart', (tag) => {
charData.openTag(tag.name);
currentTag = tag.name;
});
this.saxStream.on('opentag', (tag) => {
if (!config.isKnownTag(tag.name)) {
this.logger('warn', 'unhandled tag', tag.name);
this.err(`unhandled tag: ${tag.name}`);
}
this.onOpenTag(tag);
});
this.saxStream.on('text', charData.append);
this.saxStream.on('cdata', charData.append);
this.saxStream.on('attribute', (attr) => {
this.onAttribute(currentTag, attr);
});
this.saxStream.on('closetag', (tag) => {
charData.closeTag();
this.onCloseTag(tag);
});
}
/** The first parse error, or null if there has been none */
get error() {
return this.errors[0] ?? null;
}
/** Called for every opening tag, after unknown tags have been reported */
onOpenTag(tag) { }
/** Count a finished entry and push it, unless it is past the entry cap */
emitEntry(item) {
this.entryCount++;
if (this.entryCount > this.maxEntries) {
// Report once rather than for every dropped entry
if (this.entryCount === this.maxEntries + 1) {
const msg = this.overflowMessage(this.maxEntries);
this.err(msg);
// A listener (such as parseSitemapIndex) handles the overflow
// itself; only log when nobody is listening
if (!this.emit('maxEntriesExceeded', this.maxEntries)) {
this.logger('error', msg);
}
}
}
else {
this.push(item);
}
}
_transform(data, encoding, callback) {
try {
const cb = () => callback(this.level === ErrorLevel.THROW ? this.error : null);
// correcting the type here can be done without making it a breaking change
// TODO fix this
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
if (!this.saxStream.write(data, encoding)) {
this.saxStream.once('drain', cb);
}
else {
process.nextTick(cb);
}
}
catch (error) {
callback(error);
}
}
/**
* Tell sax the document has ended, so a truncated document (an unclosed
* element) fails the stream like any other malformed XML.
*/
_flush(callback) {
// Past the entry cap the rest of the document is ignored, and a caller
// such as parseSitemap may have stopped reading the source on purpose,
// so a missing end is not an error then.
if (this.entryCount > this.maxEntries) {
callback();
return;
}
try {
this.saxStream.end();
callback();
}
catch (error) {
callback(error);
}
}
/** Log a message and record it as a parse error, with the same text */
report(level, msg) {
this.logger(level, msg);
this.err(msg);
}
/** Record a parse error, keeping at most LIMITS.MAX_PARSER_ERRORS */
err(msg) {
this.errorCount++;
if (this.errors.length < LIMITS.MAX_PARSER_ERRORS) {
this.errors.push(new Error(msg));
}
}
}
/**
* Pipe `xml` through `parser` and resolve with every entry it emits. Rejects
* if either stream fails.
*
* Entries past the parser's cap are dropped, so once the cap is crossed this
* stops reading `xml` and destroys it instead of parsing the rest (BB-05).
* `onOverflow` then picks the outcome: return an Error to reject with it, or
* nothing to resolve with the entries emitted before the cap.
*/
export function collectEntries(xml, parser, onOverflow) {
return new Promise((resolve, reject) => {
const items = [];
let settled = false;
const settle = (error) => {
if (settled)
return;
settled = true;
if (error) {
reject(error);
}
else {
resolve(items);
}
};
// pipe() does not forward source errors, so watch the source too
xml.on('error', (error) => {
settle(error);
parser.destroy();
});
parser.once('maxEntriesExceeded', (max) => {
xml.unpipe(parser);
xml.destroy();
const error = onOverflow(max);
if (error) {
settle(error);
parser.destroy();
}
else {
// Ending the parser still delivers the entries it already emitted
parser.end();
}
});
parser
.on('data', (item) => {
if (!settled)
items.push(item);
})
.on('end', () => settle())
.on('error', (error) => settle(error));
xml.pipe(parser);
});
}
+56
View File
@@ -0,0 +1,56 @@
import type { QualifiedAttribute } from 'sax';
import { Readable, Transform, type TransformOptions, type TransformCallback } from 'node:stream';
import { type IndexItem, type XMLToSitemapIndexItemStreamOptions } from './types.js';
import { SaxParserStream } from './sax-parser-stream.js';
export type { XMLToSitemapIndexItemStreamOptions };
/**
* Takes a stream of xml and transforms it into a stream of IndexItems
* Use this to parse existing sitemap indices into config options compatible with this library
*
* Emits `'maxEntriesExceeded'` (with the cap) once, when the first entry past
* `maxEntries` is seen. If nothing listens for that event, the stream logs the
* overflow through its logger instead.
*/
export declare class XMLToSitemapIndexStream extends SaxParserStream<IndexItem> {
private currentItem;
constructor(opts?: XMLToSitemapIndexItemStreamOptions);
protected onCharData(tag: string, text: string): void;
protected onAttribute(currentTag: string, attr: QualifiedAttribute): void;
protected onCloseTag(tag: string): void;
}
/**
Read xml and resolve with the configuration that would produce it or reject with
an error
```
import { createReadStream } from 'fs'
import { Readable } from 'stream'
import { parseSitemapIndex, SitemapIndexStream, streamToPromise } from 'sitemap'
const items = await parseSitemapIndex(createReadStream('./example-index.xml'))
// produces the same xml
// you can, of course, more practically modify it or store it
const smis = new SitemapIndexStream()
Readable.from(items).pipe(smis)
console.log((await streamToPromise(smis)).toString())
```
@param {Readable} xml what to parse
@param {number} maxEntries Maximum number of sitemap entries to parse (default: 50,000 per sitemaps.org spec).
Values above 50,000 have no effect: the spec caps a sitemap index at 50,000 entries, and larger sets must be split.
0 rejects on the first entry. Anything other than a non-negative integer or Infinity rejects immediately.
@return {Promise<IndexItem[]>} resolves with list of index items that can be fed into a SitemapIndexStream. Rejects with an Error object.
*/
export declare function parseSitemapIndex(xml: Readable, maxEntries?: number): Promise<IndexItem[]>;
export interface IndexObjectStreamToJSONOptions extends TransformOptions {
lineSeparated: boolean;
}
/**
* A Transform that converts a stream of objects into a JSON Array or a line
* separated stringified JSON
* @param [lineSeparated=false] whether to separate entries by a new line or comma
*/
export declare class IndexObjectStreamToJSON extends Transform {
lineSeparated: boolean;
firstWritten: boolean;
constructor(opts?: IndexObjectStreamToJSONOptions);
_transform(chunk: IndexItem, encoding: string, cb: TransformCallback): void;
_flush(cb: TransformCallback): void;
}
+155
View File
@@ -0,0 +1,155 @@
import { Readable, Transform, } from 'node:stream';
import { IndexTagNames, } from './types.js';
import { getURLError, isValidISODate, isValidMaxEntries, } from './validation.js';
import { LIMITS } from './constants.js';
import { SaxParserStream, collectEntries } from './sax-parser-stream.js';
function isValidTagName(tagName) {
// This only works because the enum name and value are the same
return tagName in IndexTagNames;
}
function tagTemplate() {
return {
url: '',
};
}
const defaultStreamOpts = {};
/**
* Takes a stream of xml and transforms it into a stream of IndexItems
* Use this to parse existing sitemap indices into config options compatible with this library
*
* Emits `'maxEntriesExceeded'` (with the cap) once, when the first entry past
* `maxEntries` is seen. If nothing listens for that event, the stream logs the
* overflow through its logger instead.
*/
export class XMLToSitemapIndexStream extends SaxParserStream {
currentItem = tagTemplate();
constructor(opts = defaultStreamOpts) {
super(opts, {
isKnownTag: isValidTagName,
overflowMessage: (max) => `Sitemap index exceeds maximum of ${max} entries`,
});
}
onCharData(tag, text) {
switch (tag) {
case IndexTagNames.loc:
// Validate URL for security: prevents protocol injection, checks length limits
{
const errMsg = getURLError(text, 'Sitemap index URL');
if (errMsg) {
this.report('warn', `Invalid URL in sitemap index: ${errMsg}`);
}
else {
this.currentItem.url = text;
}
}
break;
case IndexTagNames.lastmod:
// Validate date format for security and spec compliance
if (!isValidISODate(text)) {
this.report('warn', `Invalid lastmod date format in sitemap index: ${text}`);
}
else {
this.currentItem.lastmod = text;
}
break;
default:
this.logger('log', 'unhandled text for tag:', tag, `'${text}'`);
this.err(`unhandled text for tag: ${tag} '${text}'`);
break;
}
}
onAttribute(currentTag, attr) {
switch (currentTag) {
case IndexTagNames.sitemapindex:
break;
default:
this.logger('log', 'unhandled attr', currentTag, attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
}
onCloseTag(tag) {
if (tag === IndexTagNames.sitemap) {
// Only emit items with valid URLs (non-empty after validation). Invalid
// ones were already reported and don't count toward maxEntries.
if (this.currentItem.url) {
this.emitEntry(this.currentItem);
}
this.currentItem = tagTemplate();
}
}
}
/**
Read xml and resolve with the configuration that would produce it or reject with
an error
```
import { createReadStream } from 'fs'
import { Readable } from 'stream'
import { parseSitemapIndex, SitemapIndexStream, streamToPromise } from 'sitemap'
const items = await parseSitemapIndex(createReadStream('./example-index.xml'))
// produces the same xml
// you can, of course, more practically modify it or store it
const smis = new SitemapIndexStream()
Readable.from(items).pipe(smis)
console.log((await streamToPromise(smis)).toString())
```
@param {Readable} xml what to parse
@param {number} maxEntries Maximum number of sitemap entries to parse (default: 50,000 per sitemaps.org spec).
Values above 50,000 have no effect: the spec caps a sitemap index at 50,000 entries, and larger sets must be split.
0 rejects on the first entry. Anything other than a non-negative integer or Infinity rejects immediately.
@return {Promise<IndexItem[]>} resolves with list of index items that can be fed into a SitemapIndexStream. Rejects with an Error object.
*/
export async function parseSitemapIndex(xml, maxEntries = LIMITS.MAX_SITEMAP_ITEM_LIMIT) {
if (!isValidMaxEntries(maxEntries)) {
// Release the source before rejecting; nothing else will read it
xml.destroy();
throw new Error(`maxEntries must be a non-negative integer or Infinity, got ${maxEntries}`);
}
// The spec caps an index at 50,000 entries, so a larger maxEntries has no
// effect. The stream drops entries past this cap instead of pushing them.
const cap = Math.min(maxEntries, LIMITS.MAX_SITEMAP_ITEM_LIMIT);
// Listening for the overflow also stops the parser logging it, since this
// rejection already reports it.
return collectEntries(xml, new XMLToSitemapIndexStream({ maxEntries: cap }), () => new Error(`Sitemap index exceeds maximum allowed entries (${cap})`));
}
const defaultObjectStreamOpts = {
lineSeparated: false,
};
/**
* A Transform that converts a stream of objects into a JSON Array or a line
* separated stringified JSON
* @param [lineSeparated=false] whether to separate entries by a new line or comma
*/
export class IndexObjectStreamToJSON extends Transform {
lineSeparated;
firstWritten;
constructor(opts = defaultObjectStreamOpts) {
opts.writableObjectMode = true;
super(opts);
this.lineSeparated = opts.lineSeparated;
this.firstWritten = false;
}
_transform(chunk, encoding, cb) {
if (!this.firstWritten) {
this.firstWritten = true;
if (!this.lineSeparated) {
this.push('[');
}
}
else if (this.lineSeparated) {
this.push('\n');
}
else {
this.push(',');
}
if (chunk) {
this.push(JSON.stringify(chunk));
}
cb();
}
_flush(cb) {
if (!this.lineSeparated) {
this.push(']');
}
cb();
}
}
+206
View File
@@ -0,0 +1,206 @@
import { WriteStream } from 'node:fs';
import { Transform, type TransformOptions, type TransformCallback } from 'node:stream';
import { type IndexItem, type SitemapItemLoose, type Logger, ErrorLevel, IndexTagNames } from './types.js';
import { SitemapStream } from './sitemap-stream.js';
export { IndexTagNames };
/**
* Options for the SitemapIndexStream
*/
export interface SitemapIndexStreamOptions extends TransformOptions {
/**
* Whether to output the lastmod date only (no time)
*
* @default false
*/
lastmodDateOnly?: boolean;
/**
* How to handle errors in passed in urls
*
* @default ErrorLevel.WARN
*/
level?: ErrorLevel;
/**
* URL to an XSL stylesheet to include in the XML
*/
xslUrl?: string;
/**
* Receives warnings about invalid index items. Defaults to the console;
* pass `false` to silence.
*/
logger?: Logger | false;
}
/**
* `SitemapIndexStream` is a Transform stream that takes `IndexItem`s or sitemap URL strings and outputs a stream of sitemap index XML.
*
* It automatically handles the XML declaration and the opening and closing tags for the sitemap index.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @extends {Transform}
*/
export declare class SitemapIndexStream extends Transform {
lastmodDateOnly: boolean;
level: ErrorLevel;
xslUrl?: string;
logger: Logger;
private hasHeadOutput;
/**
* `SitemapIndexStream` is a Transform stream that takes `IndexItem`s or sitemap URL strings and outputs a stream of sitemap index XML.
*
* It automatically handles the XML declaration and the opening and closing tags for the sitemap index.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @param {SitemapIndexStreamOptions} [opts=defaultStreamOpts] - Stream options.
*/
constructor(opts?: SitemapIndexStreamOptions);
/** Log a warning unless the current level is SILENT */
private warn;
private writeHeadOutput;
_transform(item: IndexItem | string, encoding: string, callback: TransformCallback): void;
_flush(cb: TransformCallback): void;
}
/**
* Callback function type for creating new sitemap streams when the item limit is reached.
*
* This function is called by SitemapAndIndexStream to create a new sitemap file when
* the current one reaches the item limit.
*
* @param i - The zero-based index of the sitemap file being created (0 for first sitemap,
* 1 for second, etc.)
* @returns A tuple containing:
* - [0]: IndexItem or URL string to add to the sitemap index
* - [1]: SitemapStream instance for writing sitemap items
* - [2]: WriteStream where the sitemap will be piped (the stream will be
* awaited for 'finish' before creating the next sitemap)
*
* If other streams sit between the sitemap and the WriteStream, such as
* `createGzip()`, connect them with `pipeline()` from `node:stream` rather than
* `pipe()`, so a failure in any of them also fails the WriteStream.
*
* The index item is checked only after this callback returns. If it is
* rejected, the returned streams are destroyed, but a file the callback has
* already created stays on disk, so validate the URL (for example with
* `validateURL`) before opening the file.
*
* @example
* ```typescript
* const getSitemapStream = (i: number) => {
* const sitemapStream = new SitemapStream();
* const path = `./sitemap-${i}.xml`;
* const writeStream = createWriteStream(path);
* sitemapStream.pipe(writeStream);
* return [`https://example.com/${path}`, sitemapStream, writeStream];
* };
* ```
*/
type getSitemapStreamFunc = (i: number) => [IndexItem | string, SitemapStream, WriteStream];
/**
* Options for the SitemapAndIndexStream
*
* @extends {SitemapIndexStreamOptions}
*/
export interface SitemapAndIndexStreamOptions extends SitemapIndexStreamOptions {
/**
* Max number of items in each sitemap XML file.
*
* When the limit is reached the current sitemap file will be closed,
* a wait for `finish` on the target write stream will happen,
* and a new sitemap file will be created.
*
* Range: 1 - 50,000
*
* @default 45000
*/
limit?: number;
/**
* Callback for SitemapIndexAndStream that creates a new sitemap stream for a given sitemap index.
*
* Called when a new sitemap file is needed.
*
* The write stream is the destination where the sitemap was piped.
* SitemapAndIndexStream will wait for the `finish` event on each sitemap's
* write stream before moving on to the next sitemap. This ensures that the
* contents of the write stream will be fully written before being used
* by any following operations (e.g. uploading, reading contents for unit tests).
*
* @param i - The index of the sitemap file
* @returns A tuple containing the index item to be written into the sitemap index, the sitemap stream, and the write stream for the sitemap pipe destination
*/
getSitemapStream: getSitemapStreamFunc;
}
/**
* `SitemapAndIndexStream` is a Transform stream that takes in sitemap items,
* writes them to sitemap files, adds the sitemap files to a sitemap index,
* and creates new sitemap files when the count limit is reached.
*
* It waits for the target stream of the current sitemap file to finish before
* moving on to the next if the target stream is returned by the `getSitemapStream`
* callback in the 3rd position of the tuple.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @extends {SitemapIndexStream}
*/
export declare class SitemapAndIndexStream extends SitemapIndexStream {
private itemsWritten;
private getSitemapStream;
private limit;
/**
* The open sitemap, its pipe target, if any, and a promise that settles
* once both have finished. The promise rejects if either errors or is
* destroyed first.
*/
private current?;
/**
* `SitemapAndIndexStream` is a Transform stream that takes in sitemap items,
* writes them to sitemap files, adds the sitemap files to a sitemap index,
* and creates new sitemap files when the count limit is reached.
*
* It waits for the target stream of the current sitemap file to finish before
* moving on to the next if the target stream is returned by the `getSitemapStream`
* callback in the 3rd position of the tuple.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @param {SitemapAndIndexStreamOptions} opts - Stream options.
*/
constructor(opts: SitemapAndIndexStreamOptions);
_transform(item: SitemapItemLoose, encoding: string, callback: TransformCallback): void;
/**
* Ends the current sitemap and waits until it and its pipe target, if any,
* have finished, so each sitemap file is fully written before the next one
* starts. Rejects if either stream errors or is destroyed before finishing.
*/
private closeCurrentSitemap;
private writeItem;
/**
* Destroys the open sitemap and its target, if any, so a stopped stream
* does not leave a file descriptor open.
*/
_destroy(err: Error | null, cb: (error?: Error | null) => void): void;
/**
* Called when the stream is finished.
* Closes the last sitemap before writing the end of the sitemap index.
*
* @param cb - The callback to invoke when flushing is complete
*/
_flush(cb: TransformCallback): void;
/**
* Asks `getSitemapStream` for the next sitemap, adds it to the index and
* makes it current. Throws if the callback throws or returns an invalid
* tuple, or if the index rejects the sitemap's index item, after destroying
* any streams the callback returned.
*/
private createSitemap;
/**
* Validates the rest of the tuple `getSitemapStream` returned and adds the
* index item to the index. Throws if any element is invalid, or if the
* index rejects the index item.
*/
private addToIndex;
}
+368
View File
@@ -0,0 +1,368 @@
import { WriteStream } from 'node:fs';
import { Transform, } from 'node:stream';
import { finished } from 'node:stream/promises';
import { ErrorLevel, IndexTagNames, } from './types.js';
import { resolveLogger } from './logger.js';
import { errorMessage } from './utils.js';
import { SitemapStream, stylesheetInclude } from './sitemap-stream.js';
import { element, otag, ctag } from './sitemap-xml.js';
import { LIMITS, DEFAULT_SITEMAP_ITEM_LIMIT } from './constants.js';
import { validateURL, validateXSLUrl } from './validation.js';
import { toError } from './utils.js';
// Re-export IndexTagNames for backward compatibility
export { IndexTagNames };
const xmlDec = '<?xml version="1.0" encoding="UTF-8"?>';
const sitemapIndexTagStart = '<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">';
const closetag = '</sitemapindex>';
const defaultStreamOpts = {};
/**
* `SitemapIndexStream` is a Transform stream that takes `IndexItem`s or sitemap URL strings and outputs a stream of sitemap index XML.
*
* It automatically handles the XML declaration and the opening and closing tags for the sitemap index.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @extends {Transform}
*/
export class SitemapIndexStream extends Transform {
lastmodDateOnly;
level;
xslUrl;
logger;
hasHeadOutput;
/**
* `SitemapIndexStream` is a Transform stream that takes `IndexItem`s or sitemap URL strings and outputs a stream of sitemap index XML.
*
* It automatically handles the XML declaration and the opening and closing tags for the sitemap index.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @param {SitemapIndexStreamOptions} [opts=defaultStreamOpts] - Stream options.
*/
constructor(opts = defaultStreamOpts) {
opts.objectMode = true;
super(opts);
this.hasHeadOutput = false;
this.lastmodDateOnly = opts.lastmodDateOnly || false;
this.level = opts.level ?? ErrorLevel.WARN;
// Resolve without the level: `level` is a public field that callers may
// change after construction, so warn() checks it at call time instead
this.logger = resolveLogger(ErrorLevel.WARN, opts.logger);
if (opts.xslUrl !== undefined) {
validateXSLUrl(opts.xslUrl);
}
this.xslUrl = opts.xslUrl;
}
/** Log a warning unless the current level is SILENT */
warn(...message) {
if (this.level !== ErrorLevel.SILENT) {
this.logger('warn', ...message);
}
}
writeHeadOutput() {
this.hasHeadOutput = true;
let stylesheet = '';
if (this.xslUrl) {
stylesheet = stylesheetInclude(this.xslUrl);
}
this.push(xmlDec + stylesheet + sitemapIndexTagStart);
}
_transform(item, encoding, callback) {
if (!this.hasHeadOutput) {
this.writeHeadOutput();
}
try {
// Validate URL using centralized validation (checks protocol, length, format)
const url = typeof item === 'string' ? item : item.url;
if (!url || typeof url !== 'string') {
const error = new Error('Invalid sitemap index item: URL must be a non-empty string');
if (this.level === ErrorLevel.THROW) {
callback(error);
return;
}
else {
this.warn(error.message, item);
}
// The logger is a no-op at SILENT; skip this item
callback();
return;
}
// Security: Use centralized validation to enforce protocol restrictions,
// length limits, and prevent injection attacks
try {
validateURL(url, 'Sitemap index URL');
}
catch (error) {
// Wrap the validation error with consistent message format
const validationMsg = error instanceof Error ? error.message : String(error);
const err = new Error(`Invalid URL in sitemap index: ${validationMsg}`);
if (this.level === ErrorLevel.THROW) {
callback(err);
return;
}
else {
this.warn(err.message);
}
// The logger is a no-op at SILENT; skip this item
callback();
return;
}
// Check lastmod before pushing anything, so a rejected item leaves no
// partial <sitemap> element in the output.
let lastmod;
if (typeof item !== 'string' && item.lastmod) {
try {
lastmod = new Date(item.lastmod).toISOString();
if (this.lastmodDateOnly) {
lastmod = lastmod.slice(0, 10);
}
}
catch {
const error = new Error(`Invalid lastmod date in sitemap index: ${item.lastmod}`);
if (this.level === ErrorLevel.THROW) {
callback(error);
return;
}
else {
this.warn(error.message);
}
// The logger is a no-op at SILENT; continue without lastmod
}
}
this.push(otag(IndexTagNames.sitemap));
this.push(element(IndexTagNames.loc, url));
if (lastmod) {
this.push(element(IndexTagNames.lastmod, lastmod));
}
this.push(ctag(IndexTagNames.sitemap));
callback();
}
catch (error) {
callback(toError(error));
}
}
_flush(cb) {
if (!this.hasHeadOutput) {
this.writeHeadOutput();
}
this.push(closetag);
cb();
}
}
/**
* `SitemapAndIndexStream` is a Transform stream that takes in sitemap items,
* writes them to sitemap files, adds the sitemap files to a sitemap index,
* and creates new sitemap files when the count limit is reached.
*
* It waits for the target stream of the current sitemap file to finish before
* moving on to the next if the target stream is returned by the `getSitemapStream`
* callback in the 3rd position of the tuple.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @extends {SitemapIndexStream}
*/
export class SitemapAndIndexStream extends SitemapIndexStream {
itemsWritten;
getSitemapStream;
limit;
/**
* The open sitemap, its pipe target, if any, and a promise that settles
* once both have finished. The promise rejects if either errors or is
* destroyed first.
*/
current;
/**
* `SitemapAndIndexStream` is a Transform stream that takes in sitemap items,
* writes them to sitemap files, adds the sitemap files to a sitemap index,
* and creates new sitemap files when the count limit is reached.
*
* It waits for the target stream of the current sitemap file to finish before
* moving on to the next if the target stream is returned by the `getSitemapStream`
* callback in the 3rd position of the tuple.
*
* ⚠️ CAUTION: This object is `readable` and must be read (e.g. piped to a file or to /dev/null)
* before `finish` will be emitted. Failure to read the stream will result in hangs.
*
* @param {SitemapAndIndexStreamOptions} opts - Stream options.
*/
constructor(opts) {
opts.objectMode = true;
super(opts);
this.itemsWritten = 0;
this.getSitemapStream = opts.getSitemapStream;
this.limit = opts.limit ?? DEFAULT_SITEMAP_ITEM_LIMIT;
// Validate limit is within acceptable range per sitemaps.org spec
// See: https://www.sitemaps.org/protocol.html#index
if (this.limit < LIMITS.MIN_SITEMAP_ITEM_LIMIT ||
this.limit > LIMITS.MAX_SITEMAP_ITEM_LIMIT) {
throw new Error(`limit must be between ${LIMITS.MIN_SITEMAP_ITEM_LIMIT} and ${LIMITS.MAX_SITEMAP_ITEM_LIMIT} per sitemaps.org spec, got ${this.limit}`);
}
}
_transform(item, encoding, callback) {
if (this.current && this.itemsWritten % this.limit !== 0) {
this.writeItem(this.current.sitemap, item, callback);
return;
}
const startNextSitemap = () => {
// The stream may have been destroyed while the last sitemap closed.
if (this.destroyed) {
return;
}
// Catch here rather than relying on the caller: after rotation this
// runs inside a promise callback, where a throw would become an
// unhandled rejection.
try {
this.writeItem(this.createSitemap(encoding), item, callback);
}
catch (err) {
callback(err);
}
};
if (!this.current) {
startNextSitemap();
return;
}
// Transform delivers items one at a time, so no other item can
// arrive until this rotation calls back.
this.closeCurrentSitemap().then(startNextSitemap, callback);
}
/**
* Ends the current sitemap and waits until it and its pipe target, if any,
* have finished, so each sitemap file is fully written before the next one
* starts. Rejects if either stream errors or is destroyed before finishing.
*/
async closeCurrentSitemap() {
const current = this.current;
if (!current) {
return;
}
// The pipe target gets its end() call from the sitemap stream ending.
current.sitemap.end();
await current.finished;
this.current = undefined;
}
writeItem(sitemap, item, callback) {
// SitemapStream reports a bad item (such as a relative URL with no
// hostname) through its 'error' event, which createSitemap's listener
// turns into this stream's error. The sitemap comes from
// getSitemapStream, though, so a caller's subclass or wrapper may still
// throw from write(); report that as a stream error too.
let writable;
try {
writable = sitemap.write(item);
}
catch (err) {
callback(err);
return;
}
if (!writable) {
sitemap.once('drain', callback);
}
else {
process.nextTick(callback);
}
// Increment the count of items written
this.itemsWritten++;
}
/**
* Destroys the open sitemap and its target, if any, so a stopped stream
* does not leave a file descriptor open.
*/
_destroy(err, cb) {
this.current?.sitemap.destroy();
this.current?.target?.destroy();
super._destroy(err, cb);
}
/**
* Called when the stream is finished.
* Closes the last sitemap before writing the end of the sitemap index.
*
* @param cb - The callback to invoke when flushing is complete
*/
_flush(cb) {
this.closeCurrentSitemap()
.then(() => {
super._flush(cb);
})
.catch(cb);
}
/**
* Asks `getSitemapStream` for the next sitemap, adds it to the index and
* makes it current. Throws if the callback throws or returns an invalid
* tuple, or if the index rejects the sitemap's index item, after destroying
* any streams the callback returned.
*/
createSitemap(encoding) {
const sitemapIndex = this.itemsWritten / this.limit;
let result;
try {
result = this.getSitemapStream(sitemapIndex);
}
catch (err) {
throw new Error(`getSitemapStream callback threw an error for index ${sitemapIndex}: ${errorMessage(err)}`, { cause: err });
}
// Validate the return value
if (!Array.isArray(result) || result.length !== 3) {
throw new Error(`getSitemapStream must return a 3-element array [IndexItem | string, SitemapStream, WriteStream], got: ${typeof result}`);
}
const [idxItem, sitemap, target] = result;
try {
this.addToIndex(idxItem, sitemap, target, encoding);
}
catch (err) {
// The callback has already opened these streams, so a rejected tuple
// must not leave them, or the file they write to, open.
for (const stream of [sitemap, target]) {
if (typeof stream?.destroy === 'function') {
stream.destroy();
}
}
throw err;
}
// Stays attached after rotation too, so an error a retired sitemap emits
// later is reported rather than crashing the process as unhandled.
sitemap.on('error', (err) => this.destroy(err));
// Watch both streams from the start, not only while closing, so a target
// that fails or goes away mid-sitemap stops this stream instead of
// leaving a write waiting for 'drain' or crashing on an unhandled error.
const opts = { readable: false, cleanup: true };
const finishedBoth = Promise.all([
finished(sitemap, opts),
target && finished(target, opts),
]).then(() => undefined);
finishedBoth.catch((err) => this.destroy(err));
this.current = { sitemap, target, finished: finishedBoth };
return sitemap;
}
/**
* Validates the rest of the tuple `getSitemapStream` returned and adds the
* index item to the index. Throws if any element is invalid, or if the
* index rejects the index item.
*/
addToIndex(idxItem, sitemap, target, encoding) {
if (!idxItem ||
(typeof idxItem !== 'string' && typeof idxItem !== 'object')) {
throw new Error('getSitemapStream must return an IndexItem or string as the first element');
}
if (!sitemap || typeof sitemap.write !== 'function') {
throw new Error('getSitemapStream must return a SitemapStream as the second element');
}
if (target && typeof target.write !== 'function') {
throw new Error('getSitemapStream must return a WriteStream or undefined as the third element');
}
// SitemapIndexStream._transform calls back synchronously, and at
// ErrorLevel.THROW it calls back with an error for an invalid index item,
// which fails this sitemap too.
let indexItemError;
super._transform(idxItem, encoding, (err) => {
indexItemError = err;
});
if (indexItemError) {
throw indexItemError;
}
}
}
+30
View File
@@ -0,0 +1,30 @@
import { Transform, type TransformOptions, type TransformCallback } from 'node:stream';
import { type SitemapItem, ErrorLevel } from './types.js';
/**
* Serializes one normalized sitemap item to its `<url>` element. Throws if
* the item can't be written, such as a video attribute that isn't a string.
* Drops an alternate link with no `lang` or `hreflang`.
*
* @param item - A normalized item, as `normalizeURL` returns
* @returns The `<url>…</url>` XML for the item
*/
export declare function sitemapItemToXML(item: SitemapItem): string;
export interface SitemapItemStreamOptions extends TransformOptions {
level?: ErrorLevel;
}
/**
* Takes a stream of SitemapItemOptions and spits out xml for each
* @example
* // writes <url><loc>https://example.com</loc><url><url><loc>https://example.com/2</loc><url>
* const smis = new SitemapItemStream({level: 'warn'})
* smis.pipe(writestream)
* smis.write({url: 'https://example.com', img: [], video: [], links: []})
* smis.write({url: 'https://example.com/2', img: [], video: [], links: []})
* smis.end()
* @param level - Error level
*/
export declare class SitemapItemStream extends Transform {
level: ErrorLevel;
constructor(opts?: SitemapItemStreamOptions);
_transform(item: SitemapItem, encoding: string, callback: TransformCallback): void;
}
+234
View File
@@ -0,0 +1,234 @@
import { Transform, } from 'node:stream';
import { InvalidAttr } from './errors.js';
import { ErrorLevel, TagNames, } from './types.js';
import { element, otag, ctag } from './sitemap-xml.js';
import { toError } from './utils.js';
import { getLinkHreflang } from './validation.js';
/**
* Builds an attributes object for XML elements from configuration object
* Extracts attributes based on colon-delimited keys (e.g., 'price:currency' -> { currency: value })
*
* @param conf - Configuration object containing attribute values
* @param keys - Single key or array of keys in format 'namespace:attribute'
* @returns Record of attribute names to string values (may contain non-string values from conf)
* @throws {InvalidAttr} When key format is invalid (must contain exactly one colon)
*
* @example
* attrBuilder({ 'price:currency': 'USD', 'price:type': 'rent' }, ['price:currency', 'price:type'])
* // Returns: { currency: 'USD', type: 'rent' }
*/
function attrBuilder(conf, keys) {
if (typeof keys === 'string') {
keys = [keys];
}
const iv = {};
return keys.reduce((attrs, key) => {
if (conf[key] !== undefined) {
const keyAr = key.split(':');
if (keyAr.length !== 2) {
throw new InvalidAttr(key);
}
attrs[keyAr[1]] = conf[key];
}
return attrs;
}, iv);
}
/**
* Serializes one normalized sitemap item to its `<url>` element. Throws if
* the item can't be written, such as a video attribute that isn't a string.
* Drops an alternate link with no `lang` or `hreflang`.
*
* @param item - A normalized item, as `normalizeURL` returns
* @returns The `<url>…</url>` XML for the item
*/
export function sitemapItemToXML(item) {
const parts = [];
parts.push(otag(TagNames.url));
parts.push(element(TagNames.loc, item.url));
if (item.lastmod) {
parts.push(element(TagNames.lastmod, item.lastmod));
}
if (item.changefreq) {
parts.push(element(TagNames.changefreq, item.changefreq));
}
if (item.priority !== undefined && item.priority !== null) {
// A non-number reaches here only when validation let it through (WARN
// or SILENT), so write it as given, like any other invalid value.
if (typeof item.priority !== 'number' || item.fullPrecisionPriority) {
parts.push(element(TagNames.priority, String(item.priority)));
}
else {
parts.push(element(TagNames.priority, item.priority.toFixed(1)));
}
}
item.video.forEach((video) => {
parts.push(otag(TagNames['video:video']));
parts.push(element(TagNames['video:thumbnail_loc'], video.thumbnail_loc));
parts.push(element(TagNames['video:title'], video.title));
parts.push(element(TagNames['video:description'], video.description));
if (video.content_loc) {
parts.push(element(TagNames['video:content_loc'], video.content_loc));
}
if (video.player_loc) {
parts.push(element(TagNames['video:player_loc'], attrBuilder(video, ['player_loc:autoplay', 'player_loc:allow_embed']), video.player_loc));
}
if (video.duration) {
parts.push(element(TagNames['video:duration'], video.duration.toString()));
}
if (video.expiration_date) {
parts.push(element(TagNames['video:expiration_date'], video.expiration_date));
}
if (video.rating !== undefined) {
parts.push(element(TagNames['video:rating'], video.rating.toString()));
}
if (video.view_count !== undefined) {
parts.push(element(TagNames['video:view_count'], String(video.view_count)));
}
if (video.publication_date) {
parts.push(element(TagNames['video:publication_date'], video.publication_date));
}
if (video.tag && video.tag.length > 0) {
for (const tag of video.tag) {
parts.push(element(TagNames['video:tag'], tag));
}
}
if (video.category) {
parts.push(element(TagNames['video:category'], video.category));
}
if (video.family_friendly) {
parts.push(element(TagNames['video:family_friendly'], video.family_friendly));
}
if (video.restriction) {
parts.push(element(TagNames['video:restriction'], attrBuilder(video, 'restriction:relationship'), video.restriction));
}
if (video.gallery_loc) {
parts.push(element(TagNames['video:gallery_loc'], attrBuilder(video, 'gallery_loc:title'), video.gallery_loc));
}
if (video.price) {
parts.push(element(TagNames['video:price'], attrBuilder(video, [
'price:resolution',
'price:currency',
'price:type',
]), video.price));
}
if (video.requires_subscription) {
parts.push(element(TagNames['video:requires_subscription'], video.requires_subscription));
}
if (video.uploader) {
parts.push(element(TagNames['video:uploader'], attrBuilder(video, 'uploader:info'), video.uploader));
}
if (video.platform) {
parts.push(element(TagNames['video:platform'], attrBuilder(video, 'platform:relationship'), video.platform));
}
if (video.live) {
parts.push(element(TagNames['video:live'], video.live));
}
if (video.id) {
parts.push(element(TagNames['video:id'], { type: 'url' }, video.id));
}
parts.push(ctag(TagNames['video:video']));
});
item.links.forEach((link) => {
const hreflang = getLinkHreflang(link);
// A link with no language reaches here only when validation let it
// through (WARN or SILENT). Drop it: written without hreflang, it would
// be the androidLink markup and parse back as one.
if (hreflang === undefined) {
return;
}
parts.push(element(TagNames['xhtml:link'], {
rel: 'alternate',
hreflang,
href: link.url,
}));
});
if (item.expires) {
parts.push(element(TagNames.expires, new Date(item.expires).toISOString()));
}
if (item.androidLink) {
parts.push(element(TagNames['xhtml:link'], {
rel: 'alternate',
href: item.androidLink,
}));
}
if (item.ampLink) {
parts.push(element(TagNames['xhtml:link'], {
rel: 'amphtml',
href: item.ampLink,
}));
}
if (item.news) {
parts.push(otag(TagNames['news:news']));
parts.push(otag(TagNames['news:publication']));
parts.push(element(TagNames['news:name'], item.news.publication.name));
parts.push(element(TagNames['news:language'], item.news.publication.language));
parts.push(ctag(TagNames['news:publication']));
if (item.news.access) {
parts.push(element(TagNames['news:access'], item.news.access));
}
if (item.news.genres) {
parts.push(element(TagNames['news:genres'], item.news.genres));
}
parts.push(element(TagNames['news:publication_date'], item.news.publication_date));
parts.push(element(TagNames['news:title'], item.news.title));
if (item.news.keywords) {
parts.push(element(TagNames['news:keywords'], item.news.keywords));
}
if (item.news.stock_tickers) {
parts.push(element(TagNames['news:stock_tickers'], item.news.stock_tickers));
}
parts.push(ctag(TagNames['news:news']));
}
// Image handling
item.img.forEach((image) => {
parts.push(otag(TagNames['image:image']));
parts.push(element(TagNames['image:loc'], image.url));
if (image.caption) {
parts.push(element(TagNames['image:caption'], image.caption));
}
if (image.geoLocation) {
parts.push(element(TagNames['image:geo_location'], image.geoLocation));
}
if (image.title) {
parts.push(element(TagNames['image:title'], image.title));
}
if (image.license) {
parts.push(element(TagNames['image:license'], image.license));
}
parts.push(ctag(TagNames['image:image']));
});
parts.push(ctag(TagNames.url));
return parts.join('');
}
/**
* Takes a stream of SitemapItemOptions and spits out xml for each
* @example
* // writes <url><loc>https://example.com</loc><url><url><loc>https://example.com/2</loc><url>
* const smis = new SitemapItemStream({level: 'warn'})
* smis.pipe(writestream)
* smis.write({url: 'https://example.com', img: [], video: [], links: []})
* smis.write({url: 'https://example.com/2', img: [], video: [], links: []})
* smis.end()
* @param level - Error level
*/
export class SitemapItemStream extends Transform {
level;
constructor(opts = { level: ErrorLevel.WARN }) {
opts.objectMode = true;
super(opts);
this.level = opts.level || ErrorLevel.WARN;
}
_transform(item, encoding, callback) {
// Report a serialization failure as a stream error. Thrown here, it
// would escape write(), and crash the process when the stream is piped.
let xml;
try {
xml = sitemapItemToXML(item);
}
catch (err) {
callback(toError(err));
return;
}
callback(null, xml);
}
}
+67
View File
@@ -0,0 +1,67 @@
import type { QualifiedAttribute, QualifiedTag } from 'sax';
import { Readable, Transform, type TransformOptions, type TransformCallback } from 'node:stream';
import type { SitemapItem, XMLToSitemapItemStreamOptions } from './types.js';
import { SaxParserStream } from './sax-parser-stream.js';
export type { XMLToSitemapItemStreamOptions };
/**
* Takes a stream of xml and transforms it into a stream of SitemapItems
* Use this to parse existing sitemaps into config options compatible with this library
*
* Emits `'maxEntriesExceeded'` (with the cap) once, when the first `<url>` past
* `maxEntries` is seen. If nothing listens for that event, the stream logs the
* overflow through its logger instead.
*/
export declare class XMLToSitemapItemStream extends SaxParserStream<SitemapItem> {
private currentItem;
private currentVideo;
private currentImage;
private currentLink;
private dontpushCurrentLink;
constructor(opts?: XMLToSitemapItemStreamOptions);
/**
* Number of `<url>` entries seen, including any past `maxEntries`
* @deprecated Use entryCount
*/
get urlCount(): number;
protected onOpenTag(tag: QualifiedTag): void;
protected onCharData(tag: string, text: string): void;
/** Check a yes/no value, reporting it if it is invalid */
private isYesNo;
protected onAttribute(currentTag: string, attr: QualifiedAttribute): void;
protected onCloseTag(tag: string): void;
}
/**
Read xml and resolve with the configuration that would produce it or reject with
an error
```
import { createReadStream } from 'fs'
import { Readable } from 'stream'
import { parseSitemap, SitemapStream, streamToPromise } from 'sitemap'
const items = await parseSitemap(createReadStream('./example.xml'))
// produces the same xml
// you can, of course, more practically modify it or store it
const sms = new SitemapStream()
Readable.from(items).pipe(sms)
console.log((await streamToPromise(sms)).toString())
```
Once the sitemap passes 50,000 `<url>` entries, parsing stops: `xml` is
destroyed and the promise resolves with the first 50,000 items.
@param {Readable} xml what to parse
@return {Promise<SitemapItem[]>} resolves with list of sitemap items that can be fed into a SitemapStream. Rejects with an Error object if `xml` or the parser fails, including on truncated XML.
*/
export declare function parseSitemap(xml: Readable): Promise<SitemapItem[]>;
export interface ObjectStreamToJSONOptions extends TransformOptions {
lineSeparated: boolean;
}
/**
* A Transform that converts a stream of objects into a JSON Array or a line
* separated stringified JSON
* @param [lineSeparated=false] whether to separate entries by a new line or comma
*/
export declare class ObjectStreamToJSON extends Transform {
lineSeparated: boolean;
firstWritten: boolean;
constructor(opts?: ObjectStreamToJSONOptions);
_transform(chunk: SitemapItem, encoding: string, cb: TransformCallback): void;
_flush(cb: TransformCallback): void;
}
+583
View File
@@ -0,0 +1,583 @@
import { Readable, Transform, } from 'node:stream';
import { TagNames } from './types.js';
import { isValidChangeFreq, isValidYesNo, isAllowDeny, isPriceType, isResolution, isValidPriority, isValidVideoDuration, isValidVideoRating, isValidVideoViewCount, isValidISODate, getURLError, } from './validation.js';
import { LIMITS } from './constants.js';
import { SaxParserStream, collectEntries } from './sax-parser-stream.js';
function isValidTagName(tagName) {
// This only works because the enum name and value are the same
return tagName in TagNames;
}
function getAttrValue(attr) {
if (!attr)
return undefined;
return typeof attr === 'string' ? attr : attr.value;
}
function tagTemplate() {
return {
img: [],
video: [],
links: [],
url: '',
};
}
function videoTemplate() {
return {
tag: [],
thumbnail_loc: '',
title: '',
description: '',
};
}
const imageTemplate = {
url: '',
};
const linkTemplate = {
lang: '',
url: '',
};
function newsTemplate() {
return {
publication: { name: '', language: '' },
publication_date: '',
title: '',
};
}
const defaultStreamOpts = {};
// TODO does this need to end with `options`
/**
* Takes a stream of xml and transforms it into a stream of SitemapItems
* Use this to parse existing sitemaps into config options compatible with this library
*
* Emits `'maxEntriesExceeded'` (with the cap) once, when the first `<url>` past
* `maxEntries` is seen. If nothing listens for that event, the stream logs the
* overflow through its logger instead.
*/
export class XMLToSitemapItemStream extends SaxParserStream {
currentItem = tagTemplate();
currentVideo = videoTemplate();
currentImage = { ...imageTemplate };
currentLink = { ...linkTemplate };
dontpushCurrentLink = false;
constructor(opts = defaultStreamOpts) {
super(opts, {
isKnownTag: isValidTagName,
overflowMessage: (max) => `Sitemap exceeds maximum of ${max} URLs`,
});
}
/**
* Number of `<url>` entries seen, including any past `maxEntries`
* @deprecated Use entryCount
*/
get urlCount() {
return this.entryCount;
}
onOpenTag(tag) {
if (tag.name.startsWith('news:') && !this.currentItem.news) {
this.currentItem.news = newsTemplate();
}
if (tag.name === 'xhtml:link') {
// SAX returns attributes as objects with {name, value, prefix, local, uri}
// Check if required attributes exist and have values
const rel = getAttrValue(tag.attributes.rel);
const href = getAttrValue(tag.attributes.href);
const hreflang = getAttrValue(tag.attributes.hreflang);
if (!rel || !href) {
this.report('warn', 'xhtml:link missing required rel or href attribute');
// Without a target there is no link to keep
this.dontpushCurrentLink = true;
return;
}
if (rel === 'alternate' && hreflang) {
this.currentLink.url = href;
this.currentLink.lang = hreflang;
}
else if (rel === 'alternate') {
this.dontpushCurrentLink = true;
this.currentItem.androidLink = href;
}
else if (rel === 'amphtml') {
this.dontpushCurrentLink = true;
this.currentItem.ampLink = href;
}
else {
// An unsupported rel leaves currentLink empty, so don't push it
this.dontpushCurrentLink = true;
this.logger('log', 'unhandled attr for xhtml:link', tag.attributes);
this.err(`unhandled attr for xhtml:link ${JSON.stringify(tag.attributes)}`);
}
}
}
onCharData(tag, text) {
switch (tag) {
case 'mobile:mobile':
break;
case TagNames.loc:
{
const errMsg = getURLError(text, 'Sitemap URL');
if (errMsg) {
this.logger('warn', 'Invalid URL in sitemap:', errMsg);
this.err(`Invalid URL in sitemap: ${errMsg}`);
}
else {
this.currentItem.url = text;
}
}
break;
case TagNames.changefreq:
if (isValidChangeFreq(text)) {
this.currentItem.changefreq = text;
}
break;
case TagNames.priority:
{
const priority = parseFloat(text);
if (!isValidPriority(priority)) {
this.report('warn', `Invalid priority "${text}" - must be between ${LIMITS.MIN_PRIORITY} and ${LIMITS.MAX_PRIORITY}`);
}
else {
this.currentItem.priority = priority;
}
}
break;
case TagNames.lastmod:
if (isValidISODate(text)) {
this.currentItem.lastmod = text;
}
else {
this.report('warn', `Invalid lastmod date format "${text}" - expected ISO 8601 format`);
}
break;
case TagNames['video:thumbnail_loc']:
this.currentVideo.thumbnail_loc = text;
break;
case TagNames['video:tag']:
if (this.currentVideo.tag.length < LIMITS.MAX_TAGS_PER_VIDEO) {
this.currentVideo.tag.push(text);
}
else {
this.report('warn', `video has too many tags (max ${LIMITS.MAX_TAGS_PER_VIDEO})`);
}
break;
case TagNames['video:duration']:
{
const duration = parseInt(text, 10);
if (!isValidVideoDuration(duration)) {
this.report('warn', `Invalid video duration "${text}" - must be between ${LIMITS.MIN_VIDEO_DURATION} and ${LIMITS.MAX_VIDEO_DURATION} seconds`);
}
else {
this.currentVideo.duration = duration;
}
}
break;
case TagNames['video:player_loc']:
this.currentVideo.player_loc = text;
break;
case TagNames['video:content_loc']:
this.currentVideo.content_loc = text;
break;
case TagNames['video:requires_subscription']:
if (this.isYesNo('requires_subscription', text)) {
this.currentVideo.requires_subscription = text;
}
break;
case TagNames['video:publication_date']:
if (isValidISODate(text)) {
this.currentVideo.publication_date = text;
}
else {
this.report('warn', `Invalid video publication_date format "${text}" - expected ISO 8601 format`);
}
break;
case TagNames['video:id']:
this.currentVideo.id = text;
break;
case TagNames['video:restriction']:
this.currentVideo.restriction = text;
break;
case TagNames['video:view_count']:
{
const viewCount = parseInt(text, 10);
if (!isValidVideoViewCount(viewCount)) {
this.report('warn', `Invalid video view_count "${text}" - must be a positive integer`);
}
else {
this.currentVideo.view_count = viewCount;
}
}
break;
case TagNames['video:uploader']:
this.currentVideo.uploader = text;
break;
case TagNames['video:family_friendly']:
if (this.isYesNo('family_friendly', text)) {
this.currentVideo.family_friendly = text;
}
break;
case TagNames['video:expiration_date']:
if (isValidISODate(text)) {
this.currentVideo.expiration_date = text;
}
else {
this.report('warn', `Invalid video expiration_date format "${text}" - expected ISO 8601 format`);
}
break;
case TagNames['video:platform']:
this.currentVideo.platform = text;
break;
case TagNames['video:price']:
this.currentVideo.price = text;
break;
case TagNames['video:rating']:
{
const rating = parseFloat(text);
if (!isValidVideoRating(rating)) {
this.report('warn', `Invalid video rating "${text}" - must be between ${LIMITS.MIN_VIDEO_RATING} and ${LIMITS.MAX_VIDEO_RATING}`);
}
else {
this.currentVideo.rating = rating;
}
}
break;
case TagNames['video:category']:
this.currentVideo.category = text;
break;
case TagNames['video:live']:
if (this.isYesNo('live', text)) {
this.currentVideo.live = text;
}
break;
case TagNames['video:gallery_loc']:
this.currentVideo.gallery_loc = text;
break;
case TagNames['image:loc']:
this.currentImage.url = text;
break;
case TagNames['image:geo_location']:
this.currentImage.geoLocation = text;
break;
case TagNames['image:license']:
this.currentImage.license = text;
break;
case TagNames['news:access']:
if (!this.currentItem.news) {
this.currentItem.news = newsTemplate();
}
if (text === 'Registration' || text === 'Subscription') {
this.currentItem.news.access = text;
}
else {
this.report('warn', `Invalid news:access value "${text}" - must be "Registration" or "Subscription"`);
}
break;
case TagNames['news:genres']:
if (!this.currentItem.news) {
this.currentItem.news = newsTemplate();
}
this.currentItem.news.genres = text;
break;
case TagNames['news:publication_date']:
if (!this.currentItem.news) {
this.currentItem.news = newsTemplate();
}
if (isValidISODate(text)) {
this.currentItem.news.publication_date = text;
}
else {
this.report('warn', `Invalid news publication_date format "${text}" - expected ISO 8601 format`);
}
break;
case TagNames['news:keywords']:
if (!this.currentItem.news) {
this.currentItem.news = newsTemplate();
}
this.currentItem.news.keywords = text;
break;
case TagNames['news:stock_tickers']:
if (!this.currentItem.news) {
this.currentItem.news = newsTemplate();
}
this.currentItem.news.stock_tickers = text;
break;
case TagNames['news:language']:
if (!this.currentItem.news) {
this.currentItem.news = newsTemplate();
}
this.currentItem.news.publication.language = text;
break;
case TagNames['video:title']:
if (this.currentVideo.title.length + text.length <=
LIMITS.MAX_VIDEO_TITLE_LENGTH) {
this.currentVideo.title += text;
}
else {
this.report('warn', `video title exceeds max length of ${LIMITS.MAX_VIDEO_TITLE_LENGTH}`);
}
break;
case TagNames['video:description']:
if (this.currentVideo.description.length + text.length <=
LIMITS.MAX_VIDEO_DESCRIPTION_LENGTH) {
this.currentVideo.description += text;
}
else {
this.report('warn', `video description exceeds max length of ${LIMITS.MAX_VIDEO_DESCRIPTION_LENGTH}`);
}
break;
case TagNames['news:name']:
if (!this.currentItem.news) {
this.currentItem.news = newsTemplate();
}
if (this.currentItem.news.publication.name.length + text.length <=
LIMITS.MAX_NEWS_NAME_LENGTH) {
this.currentItem.news.publication.name += text;
}
else {
this.report('warn', `news name exceeds max length of ${LIMITS.MAX_NEWS_NAME_LENGTH}`);
}
break;
case TagNames['news:title']:
if (!this.currentItem.news) {
this.currentItem.news = newsTemplate();
}
if (this.currentItem.news.title.length + text.length <=
LIMITS.MAX_NEWS_TITLE_LENGTH) {
this.currentItem.news.title += text;
}
else {
this.report('warn', `news title exceeds max length of ${LIMITS.MAX_NEWS_TITLE_LENGTH}`);
}
break;
case TagNames['image:caption']:
if (!this.currentImage.caption) {
this.currentImage.caption =
text.length <= LIMITS.MAX_IMAGE_CAPTION_LENGTH
? text
: text.substring(0, LIMITS.MAX_IMAGE_CAPTION_LENGTH);
if (text.length > LIMITS.MAX_IMAGE_CAPTION_LENGTH) {
this.report('warn', `image caption exceeds max length of ${LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
}
}
else if (this.currentImage.caption.length + text.length <=
LIMITS.MAX_IMAGE_CAPTION_LENGTH) {
this.currentImage.caption += text;
}
else {
this.report('warn', `image caption exceeds max length of ${LIMITS.MAX_IMAGE_CAPTION_LENGTH}`);
}
break;
case TagNames['image:title']:
if (!this.currentImage.title) {
this.currentImage.title =
text.length <= LIMITS.MAX_IMAGE_TITLE_LENGTH
? text
: text.substring(0, LIMITS.MAX_IMAGE_TITLE_LENGTH);
if (text.length > LIMITS.MAX_IMAGE_TITLE_LENGTH) {
this.report('warn', `image title exceeds max length of ${LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
}
}
else if (this.currentImage.title.length + text.length <=
LIMITS.MAX_IMAGE_TITLE_LENGTH) {
this.currentImage.title += text;
}
else {
this.report('warn', `image title exceeds max length of ${LIMITS.MAX_IMAGE_TITLE_LENGTH}`);
}
break;
default:
this.logger('log', 'unhandled text for tag:', tag, `'${text}'`);
this.err(`unhandled text for tag: ${tag} '${text}'`);
break;
}
}
/** Check a yes/no value, reporting it if it is invalid */
isYesNo(field, value) {
if (isValidYesNo(value)) {
return true;
}
this.report('warn', `Invalid video ${field} "${value}" - expected yes or no`);
return false;
}
onAttribute(currentTag, attr) {
switch (currentTag) {
case TagNames['urlset']:
case TagNames['xhtml:link']:
case TagNames['video:id']:
break;
case TagNames['video:restriction']:
if (attr.name === 'relationship' && isAllowDeny(attr.value)) {
this.currentVideo['restriction:relationship'] = attr.value;
}
else {
this.logger('log', 'unhandled attr', currentTag, attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
break;
case TagNames['video:price']:
if (attr.name === 'type' && isPriceType(attr.value)) {
this.currentVideo['price:type'] = attr.value;
}
else if (attr.name === 'currency') {
this.currentVideo['price:currency'] = attr.value;
}
else if (attr.name === 'resolution' && isResolution(attr.value)) {
this.currentVideo['price:resolution'] = attr.value;
}
else {
this.logger('log', 'unhandled attr for video:price', attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
break;
case TagNames['video:player_loc']:
if (attr.name === 'autoplay') {
this.currentVideo['player_loc:autoplay'] = attr.value;
}
else if (attr.name === 'allow_embed') {
const allowEmbed = attr.value.trim();
if (this.isYesNo('allow_embed', allowEmbed)) {
this.currentVideo['player_loc:allow_embed'] = allowEmbed;
}
}
else {
this.logger('log', 'unhandled attr for video:player_loc', attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
break;
case TagNames['video:platform']:
if (attr.name === 'relationship' && isAllowDeny(attr.value)) {
this.currentVideo['platform:relationship'] = attr.value;
}
else {
this.logger('log', 'unhandled attr for video:platform', attr.name, attr.value);
this.err(`unhandled attr: ${currentTag} ${attr.name} ${attr.value}`);
}
break;
case TagNames['video:gallery_loc']:
if (attr.name === 'title') {
this.currentVideo['gallery_loc:title'] = attr.value;
}
else {
this.logger('log', 'unhandled attr for video:galler_loc', attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
break;
case TagNames['video:uploader']:
if (attr.name === 'info') {
this.currentVideo['uploader:info'] = attr.value;
}
else {
this.logger('log', 'unhandled attr for video:uploader', attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
break;
default:
this.logger('log', 'unhandled attr', currentTag, attr.name);
this.err(`unhandled attr: ${currentTag} ${attr.name}`);
}
}
onCloseTag(tag) {
switch (tag) {
case TagNames.url:
this.emitEntry(this.currentItem);
this.currentItem = tagTemplate();
break;
case TagNames['video:video']:
if (this.currentItem.video.length < LIMITS.MAX_VIDEOS_PER_URL) {
this.currentItem.video.push(this.currentVideo);
}
else {
this.report('warn', `URL has too many videos (max ${LIMITS.MAX_VIDEOS_PER_URL})`);
}
this.currentVideo = videoTemplate();
break;
case TagNames['image:image']:
if (this.currentItem.img.length < LIMITS.MAX_IMAGES_PER_URL) {
this.currentItem.img.push(this.currentImage);
}
else {
this.report('warn', `URL has too many images (max ${LIMITS.MAX_IMAGES_PER_URL})`);
}
this.currentImage = { ...imageTemplate };
break;
case TagNames['xhtml:link']:
if (!this.dontpushCurrentLink) {
if (this.currentItem.links.length < LIMITS.MAX_LINKS_PER_URL) {
this.currentItem.links.push(this.currentLink);
}
else {
this.report('warn', `URL has too many links (max ${LIMITS.MAX_LINKS_PER_URL})`);
}
}
this.currentLink = { ...linkTemplate };
this.dontpushCurrentLink = false; // Reset flag for next link
break;
default:
break;
}
}
}
/**
Read xml and resolve with the configuration that would produce it or reject with
an error
```
import { createReadStream } from 'fs'
import { Readable } from 'stream'
import { parseSitemap, SitemapStream, streamToPromise } from 'sitemap'
const items = await parseSitemap(createReadStream('./example.xml'))
// produces the same xml
// you can, of course, more practically modify it or store it
const sms = new SitemapStream()
Readable.from(items).pipe(sms)
console.log((await streamToPromise(sms)).toString())
```
Once the sitemap passes 50,000 `<url>` entries, parsing stops: `xml` is
destroyed and the promise resolves with the first 50,000 items.
@param {Readable} xml what to parse
@return {Promise<SitemapItem[]>} resolves with list of sitemap items that can be fed into a SitemapStream. Rejects with an Error object if `xml` or the parser fails, including on truncated XML.
*/
export async function parseSitemap(xml) {
const parser = new XMLToSitemapItemStream();
return collectEntries(xml, parser, (max) => {
parser.logger('error', `Sitemap exceeds maximum of ${max} URLs; ignoring the rest`);
});
}
const defaultObjectStreamOpts = {
lineSeparated: false,
};
/**
* A Transform that converts a stream of objects into a JSON Array or a line
* separated stringified JSON
* @param [lineSeparated=false] whether to separate entries by a new line or comma
*/
export class ObjectStreamToJSON extends Transform {
lineSeparated;
firstWritten;
constructor(opts = defaultObjectStreamOpts) {
opts.writableObjectMode = true;
super(opts);
this.lineSeparated = opts.lineSeparated;
this.firstWritten = false;
}
_transform(chunk, encoding, cb) {
if (!this.firstWritten) {
this.firstWritten = true;
if (!this.lineSeparated) {
this.push('[');
}
}
else if (this.lineSeparated) {
this.push('\n');
}
else {
this.push(',');
}
if (chunk) {
this.push(JSON.stringify(chunk));
}
cb();
}
_flush(cb) {
if (!this.lineSeparated) {
this.push(']');
}
cb();
}
}
+68
View File
@@ -0,0 +1,68 @@
import { Readable } from 'node:stream';
import type { SitemapItemLoose, Logger } from './types.js';
/**
* Options for the simpleSitemapAndIndex function
*/
export interface SimpleSitemapAndIndexOptions {
/**
* The hostname for all URLs
* Must be a valid http:// or https:// URL
*/
hostname: string;
/**
* The hostname for the sitemaps if different than hostname
* Must be a valid http:// or https:// URL
*/
sitemapHostname?: string;
/**
* The urls you want to make a sitemap out of.
* Can be an array of items, a file path string, a Readable stream, or an array of strings
*/
sourceData: SitemapItemLoose[] | string | Readable | string[];
/**
* Where to write the sitemaps and index
* Must be a relative path without path traversal sequences
*/
destinationDir: string;
/**
* Where the sitemaps are relative to the hostname. Defaults to root.
* Must not contain path traversal sequences
*/
publicBasePath?: string;
/**
* How many URLs to write before switching to a new file
* Must be between 1 and 50,000 per sitemaps.org spec
* @default 45000
*/
limit?: number;
/**
* Whether to compress the written files
* @default true
*/
gzip?: boolean;
/**
* Optional URL to an XSL stylesheet
* Must be a valid http:// or https:// URL
*/
xslUrl?: string;
/**
* Receives validation warnings from the sitemap and index streams.
* Defaults to the console; pass `false` to silence.
*/
logger?: Logger | false;
}
/**
* A simpler interface for creating sitemaps and indexes.
* Automatically handles splitting large datasets into multiple sitemap files.
*
* @param options - Configuration options
* @returns A promise that resolves when all sitemaps and the index are written
* @throws {InvalidHostnameError} If hostname or sitemapHostname is invalid
* @throws {InvalidPathError} If destinationDir contains path traversal
* @throws {InvalidPublicBasePathError} If publicBasePath is invalid
* @throws {InvalidLimitError} If limit is out of range
* @throws {InvalidXSLUrlError} If xslUrl is invalid
* @throws {Error} If sourceData type is not supported
*/
export declare const simpleSitemapAndIndex: ({ hostname, sitemapHostname, sourceData, destinationDir, limit, gzip, publicBasePath, xslUrl, logger, }: SimpleSitemapAndIndexOptions) => Promise<void>;
export default simpleSitemapAndIndex;
+101
View File
@@ -0,0 +1,101 @@
import { SitemapAndIndexStream } from './sitemap-index-stream.js';
import { SitemapStream } from './sitemap-stream.js';
import { errorMessage, lineSeparatedURLsToSitemapOptions, openSitemapFile, } from './utils.js';
import { DEFAULT_SITEMAP_ITEM_LIMIT } from './constants.js';
import { createGzip } from 'node:zlib';
import { createWriteStream, createReadStream, promises } from 'node:fs';
import { normalize, resolve } from 'node:path';
import { Readable } from 'node:stream';
import { pipeline } from 'node:stream/promises';
import { URL } from 'node:url';
import { validateURL, validatePath, validateLimit, validatePublicBasePath, validateXSLUrl, } from './validation.js';
/**
* A simpler interface for creating sitemaps and indexes.
* Automatically handles splitting large datasets into multiple sitemap files.
*
* @param options - Configuration options
* @returns A promise that resolves when all sitemaps and the index are written
* @throws {InvalidHostnameError} If hostname or sitemapHostname is invalid
* @throws {InvalidPathError} If destinationDir contains path traversal
* @throws {InvalidPublicBasePathError} If publicBasePath is invalid
* @throws {InvalidLimitError} If limit is out of range
* @throws {InvalidXSLUrlError} If xslUrl is invalid
* @throws {Error} If sourceData type is not supported
*/
export const simpleSitemapAndIndex = async ({ hostname, sitemapHostname = hostname, // if different
sourceData, destinationDir, limit = DEFAULT_SITEMAP_ITEM_LIMIT, gzip = true, publicBasePath = './', xslUrl, logger, }) => {
// Validate all inputs upfront
validateURL(hostname, 'hostname');
validateURL(sitemapHostname, 'sitemapHostname');
validatePath(destinationDir, 'destinationDir');
validateLimit(limit);
validatePublicBasePath(publicBasePath);
if (xslUrl) {
validateXSLUrl(xslUrl);
}
// Create destination directory with error context
try {
await promises.mkdir(destinationDir, { recursive: true });
}
catch (err) {
throw new Error(`Failed to create destination directory "${destinationDir}": ${errorMessage(err)}`, { cause: err });
}
// Normalize publicBasePath (don't mutate the parameter)
const normalizedPublicBasePath = publicBasePath.endsWith('/')
? publicBasePath
: publicBasePath + '/';
const sitemapAndIndexStream = new SitemapAndIndexStream({
limit,
logger,
getSitemapStream: (i) => openSitemapFile(i, new SitemapStream({ hostname, xslUrl, logger }), {
destinationDir,
gzip,
toURL: (fileName) => {
// Construct public path for the sitemap index
const publicPath = normalize(`${normalizedPublicBasePath}./${fileName}`);
try {
return new URL(publicPath, sitemapHostname).toString();
}
catch (err) {
throw new Error(`Failed to construct sitemap URL for index ${i}: ${errorMessage(err)}`, { cause: err });
}
},
urlParamName: 'sitemap URL',
onError: (err) => sitemapAndIndexStream.destroy(err),
}),
});
// Handle different sourceData types with proper error handling
let src;
if (typeof sourceData === 'string') {
try {
src = lineSeparatedURLsToSitemapOptions(createReadStream(sourceData));
}
catch (err) {
throw new Error(`Failed to read sourceData file "${sourceData}": ${errorMessage(err)}`, { cause: err });
}
}
else if (sourceData instanceof Readable) {
src = sourceData;
}
else if (Array.isArray(sourceData)) {
src = Readable.from(sourceData);
}
else {
throw new Error(`Invalid sourceData type: expected array, string (file path), or Readable stream, got ${typeof sourceData}`);
}
const writePath = resolve(destinationDir, `./sitemap-index.xml${gzip ? '.gz' : ''}`);
try {
if (gzip) {
return await pipeline(src, sitemapAndIndexStream, createGzip(), createWriteStream(writePath));
}
else {
return await pipeline(src, sitemapAndIndexStream, createWriteStream(writePath));
}
}
catch (err) {
throw new Error(`Failed to write sitemap files: ${errorMessage(err)}`, {
cause: err,
});
}
};
export default simpleSitemapAndIndex;
+89
View File
@@ -0,0 +1,89 @@
import { Transform, type TransformOptions, type TransformCallback, Readable } from 'node:stream';
import { type SitemapItemLoose, ErrorLevel, type ErrorHandler, type Logger } from './types.js';
export declare const stylesheetInclude: (url: string) => string;
export interface NSArgs {
news: boolean;
video: boolean;
xhtml: boolean;
image: boolean;
custom?: string[];
}
export declare const closetag = "</urlset>";
export interface SitemapStreamOptions extends TransformOptions {
hostname?: string;
level?: ErrorLevel;
lastmodDateOnly?: boolean;
xmlns?: NSArgs;
xslUrl?: string;
errorHandler?: ErrorHandler;
/**
* Receives validation warnings. Defaults to the console; pass `false` to
* silence. Ignored for validation when `errorHandler` is given.
*/
logger?: Logger | false;
}
/**
* A [Transform](https://nodejs.org/api/stream.html#stream_implementing_a_transform_stream)
* for turning a
* [Readable stream](https://nodejs.org/api/stream.html#stream_readable_streams)
* of either [SitemapItemOptions](#sitemap-item-options) or url strings into a
* Sitemap. The readable stream it transforms **must** be in object mode.
*
* @param {SitemapStreamOptions} opts - Configuration options
* @param {string} [opts.hostname] - Base URL for relative paths. Must use http:// or https:// protocol
* @param {ErrorLevel} [opts.level=ErrorLevel.WARN] - Error handling level (SILENT, WARN, or THROW)
* @param {boolean} [opts.lastmodDateOnly=false] - Format lastmod as date only (YYYY-MM-DD)
* @param {NSArgs} [opts.xmlns] - Control which XML namespaces to include in output
* @param {string} [opts.xslUrl] - URL to XSL stylesheet for sitemap display. Must use http:// or https://
* @param {ErrorHandler} [opts.errorHandler] - Custom error handler function
*
* @throws {InvalidHostnameError} If hostname is provided but invalid (non-http(s), malformed, or >2048 chars)
* @throws {InvalidXSLUrlError} If xslUrl is provided but invalid (non-http(s), malformed, >2048 chars, or contains malicious content)
* @throws {Error} If xmlns.custom contains invalid namespace declarations
*
* @example
* ```typescript
* const stream = new SitemapStream({
* hostname: 'https://example.com',
* level: ErrorLevel.THROW
* });
* stream.write({ url: '/page', changefreq: 'daily' });
* stream.end();
* ```
*
* @security
* - Hostname and xslUrl are validated to prevent URL injection attacks
* - Custom namespaces are validated to prevent XML injection
* - All URLs are normalized and validated before output
* - XML content is properly escaped to prevent injection
*/
export declare class SitemapStream extends Transform {
hostname?: string;
level: ErrorLevel;
hasHeadOutput: boolean;
xmlns: NSArgs;
xslUrl?: string;
errorHandler: ErrorHandler;
logger: Logger;
/**
* @deprecated Use `xmlns`
*/
get xmlNS(): NSArgs;
set xmlNS(value: NSArgs);
lastmodDateOnly: boolean;
constructor(opts?: SitemapStreamOptions);
_transform(item: SitemapItemLoose, encoding: string, callback: TransformCallback): void;
_flush(cb: TransformCallback): void;
}
/**
* Converts a readable stream into a promise that resolves with the concatenated data from the stream.
*
* The function listens for 'data' events from the stream, and when the stream ends, it resolves the promise with the concatenated data. If an error occurs while reading from the stream, the promise is rejected with the error.
*
* ⚠️ CAUTION: This function should not generally be used in production / when writing to files as it holds a copy of the entire file contents in memory until finished.
*
* @param {Readable} stream - The readable stream to convert to a promise.
* @returns {Promise<Buffer>} A promise that resolves with the concatenated data from the stream as a Buffer, or rejects with an error if one occurred while reading from the stream. If the stream is empty, the promise is rejected with an EmptyStream error.
* @throws {EmptyStream} If the stream is empty.
*/
export declare function streamToPromise(stream: Readable): Promise<Buffer>;
+240
View File
@@ -0,0 +1,240 @@
import { Transform, Readable, Writable, } from 'node:stream';
import { ErrorLevel, } from './types.js';
import { normalizeURL, toError } from './utils.js';
import { resolveLogger, loggingErrorHandler } from './logger.js';
import { validateSMIOptions, validateURL, validateXSLUrl, } from './validation.js';
import { sitemapItemToXML } from './sitemap-item-stream.js';
import { EmptyStream } from './errors.js';
import { LIMITS } from './constants.js';
const xmlDec = '<?xml version="1.0" encoding="UTF-8"?>';
export const stylesheetInclude = (url) => {
const safe = url
.replace(/&/g, '&amp;')
.replace(/"/g, '&quot;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;');
return `<?xml-stylesheet type="text/xsl" href="${safe}"?>`;
};
const urlsetTagStart = '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"';
/**
* Validates custom namespace declarations for security
* @param custom - Array of custom namespace declarations
* @throws {Error} If namespace format is invalid or contains malicious content
*/
function validateCustomNamespaces(custom) {
if (!Array.isArray(custom)) {
throw new Error('Custom namespaces must be an array');
}
// Limit number of custom namespaces to prevent DoS
if (custom.length > LIMITS.MAX_CUSTOM_NAMESPACES) {
throw new Error(`Too many custom namespaces: ${custom.length} exceeds limit of ${LIMITS.MAX_CUSTOM_NAMESPACES}`);
}
// Basic format validation for xmlns declarations and namespace-qualified attributes
// Supports both xmlns:prefix="uri" and prefix:attribute="value" (e.g., xsi:schemaLocation)
const xmlAttributePattern = /^[a-zA-Z_][\w.-]*:[a-zA-Z_][\w.-]*="[^"<>]*"$/;
for (const ns of custom) {
if (typeof ns !== 'string' || ns.length === 0) {
throw new Error('Custom namespace must be a non-empty string');
}
if (ns.length > LIMITS.MAX_NAMESPACE_LENGTH) {
throw new Error(`Custom namespace exceeds maximum length of ${LIMITS.MAX_NAMESPACE_LENGTH} characters: ${ns.substring(0, 50)}...`);
}
// Check for potentially malicious content BEFORE format check
// (format check will reject < and > but we want specific error message)
const lowerNs = ns.toLowerCase();
if (lowerNs.includes('<script') ||
lowerNs.includes('javascript:') ||
lowerNs.includes('data:text/html')) {
throw new Error(`Custom namespace contains potentially malicious content: ${ns.substring(0, 50)}`);
}
// Check format matches xmlns declaration or namespace-qualified attribute
if (!xmlAttributePattern.test(ns)) {
throw new Error(`Invalid namespace format (must be prefix:name="value", e.g., xmlns:prefix="uri" or xsi:schemaLocation="..."): ${ns.substring(0, 50)}`);
}
}
}
const getURLSetNs = ({ news, video, image, xhtml, custom }, xslURL) => {
let ns = xmlDec;
if (xslURL) {
ns += stylesheetInclude(xslURL);
}
ns += urlsetTagStart;
if (news) {
ns += ' xmlns:news="http://www.google.com/schemas/sitemap-news/0.9"';
}
if (xhtml) {
ns += ' xmlns:xhtml="http://www.w3.org/1999/xhtml"';
}
if (image) {
ns += ' xmlns:image="http://www.google.com/schemas/sitemap-image/1.1"';
}
if (video) {
ns += ' xmlns:video="http://www.google.com/schemas/sitemap-video/1.1"';
}
if (custom) {
validateCustomNamespaces(custom);
ns += ' ' + custom.join(' ');
}
return ns + '>';
};
export const closetag = '</urlset>';
const defaultXMLNS = {
news: true,
xhtml: true,
image: true,
video: true,
};
const defaultStreamOpts = {
xmlns: defaultXMLNS,
};
/**
* A [Transform](https://nodejs.org/api/stream.html#stream_implementing_a_transform_stream)
* for turning a
* [Readable stream](https://nodejs.org/api/stream.html#stream_readable_streams)
* of either [SitemapItemOptions](#sitemap-item-options) or url strings into a
* Sitemap. The readable stream it transforms **must** be in object mode.
*
* @param {SitemapStreamOptions} opts - Configuration options
* @param {string} [opts.hostname] - Base URL for relative paths. Must use http:// or https:// protocol
* @param {ErrorLevel} [opts.level=ErrorLevel.WARN] - Error handling level (SILENT, WARN, or THROW)
* @param {boolean} [opts.lastmodDateOnly=false] - Format lastmod as date only (YYYY-MM-DD)
* @param {NSArgs} [opts.xmlns] - Control which XML namespaces to include in output
* @param {string} [opts.xslUrl] - URL to XSL stylesheet for sitemap display. Must use http:// or https://
* @param {ErrorHandler} [opts.errorHandler] - Custom error handler function
*
* @throws {InvalidHostnameError} If hostname is provided but invalid (non-http(s), malformed, or >2048 chars)
* @throws {InvalidXSLUrlError} If xslUrl is provided but invalid (non-http(s), malformed, >2048 chars, or contains malicious content)
* @throws {Error} If xmlns.custom contains invalid namespace declarations
*
* @example
* ```typescript
* const stream = new SitemapStream({
* hostname: 'https://example.com',
* level: ErrorLevel.THROW
* });
* stream.write({ url: '/page', changefreq: 'daily' });
* stream.end();
* ```
*
* @security
* - Hostname and xslUrl are validated to prevent URL injection attacks
* - Custom namespaces are validated to prevent XML injection
* - All URLs are normalized and validated before output
* - XML content is properly escaped to prevent injection
*/
export class SitemapStream extends Transform {
hostname;
level;
hasHeadOutput;
xmlns;
xslUrl;
errorHandler;
logger;
/**
* @deprecated Use `xmlns`
*/
get xmlNS() {
return this.xmlns;
}
set xmlNS(value) {
this.xmlns = value;
}
lastmodDateOnly;
constructor(opts = defaultStreamOpts) {
opts.objectMode = true;
super(opts);
// Validate hostname if provided
if (opts.hostname !== undefined) {
validateURL(opts.hostname, 'hostname');
}
// Validate xslUrl if provided
if (opts.xslUrl !== undefined) {
validateXSLUrl(opts.xslUrl);
}
this.hasHeadOutput = false;
this.hostname = opts.hostname;
this.level = opts.level || ErrorLevel.WARN;
// Resolve without the level: `level` is a public field that callers may
// change after construction. validateSMIOptions receives the current
// level on every item and skips logging at SILENT.
this.logger = resolveLogger(ErrorLevel.WARN, opts.logger);
// Read this.logger at call time so reassigning it later takes effect
this.errorHandler =
opts.errorHandler ??
loggingErrorHandler((level, ...message) => this.logger(level, ...message));
this.lastmodDateOnly = opts.lastmodDateOnly || false;
this.xmlns = opts.xmlns || defaultXMLNS;
// Fail at construction, like an invalid hostname, rather than on the
// first write
if (this.xmlns.custom) {
validateCustomNamespaces(this.xmlns.custom);
}
this.xslUrl = opts.xslUrl;
}
_transform(item, encoding, callback) {
// Report every failure (an invalid item at ErrorLevel.THROW, an item
// normalizeURL rejects, one that can't be serialized) as a stream error.
// Thrown here, it would escape write(), and crash the process when the
// stream is piped.
let xml;
try {
xml = sitemapItemToXML(validateSMIOptions(normalizeURL(item, this.hostname, this.lastmodDateOnly), this.level, this.errorHandler));
if (!this.hasHeadOutput) {
this.hasHeadOutput = true;
xml = getURLSetNs(this.xmlns, this.xslUrl) + xml;
}
}
catch (err) {
callback(toError(err));
return;
}
callback(null, xml);
}
_flush(cb) {
// With no items, still emit a well-formed (empty) urlset, as
// SitemapIndexStream does for an empty index
if (!this.hasHeadOutput) {
this.hasHeadOutput = true;
this.push(getURLSetNs(this.xmlns, this.xslUrl));
}
this.push(closetag);
cb();
}
}
/**
* Converts a readable stream into a promise that resolves with the concatenated data from the stream.
*
* The function listens for 'data' events from the stream, and when the stream ends, it resolves the promise with the concatenated data. If an error occurs while reading from the stream, the promise is rejected with the error.
*
* ⚠️ CAUTION: This function should not generally be used in production / when writing to files as it holds a copy of the entire file contents in memory until finished.
*
* @param {Readable} stream - The readable stream to convert to a promise.
* @returns {Promise<Buffer>} A promise that resolves with the concatenated data from the stream as a Buffer, or rejects with an error if one occurred while reading from the stream. If the stream is empty, the promise is rejected with an EmptyStream error.
* @throws {EmptyStream} If the stream is empty.
*/
export function streamToPromise(stream) {
return new Promise((resolve, reject) => {
const drain = [];
stream
// Error propagation is not automatic
// Bubble up errors on the read stream
.on('error', reject)
.pipe(new Writable({
write(chunk, enc, next) {
drain.push(chunk);
next();
},
}))
// This bubbles up errors when writing to the internal buffer
// This is unlikely to happen, but we have this for completeness
.on('error', reject)
.on('finish', () => {
if (!drain.length) {
reject(new EmptyStream());
}
else {
resolve(Buffer.concat(drain));
}
});
});
}
+107
View File
@@ -0,0 +1,107 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
import { TagNames, IndexTagNames, type StringObj } from './types.js';
/**
* Escapes text content for safe inclusion in XML text nodes.
*
* **Security Model:**
* - Escapes `&` → `&amp;` (required to prevent entity interpretation)
* - Escapes `<` → `&lt;` (required to prevent tag injection)
* - Escapes `>` → `&gt;` (defense-in-depth, prevents CDATA injection)
* - Does NOT escape `"` or `'` (not required in text content, only in attributes)
* - Removes invalid XML Unicode characters per XML 1.0 spec
*
* **Why quotes aren't escaped:**
* In XML text content (between tags), quotes have no special meaning and don't
* need escaping. They only need escaping in attribute values, which is handled
* by the `otag()` function.
*
* @param txt - The text content to escape
* @returns XML-safe escaped text with invalid characters removed
* @throws {TypeError} If txt is not a string
*
* @example
* text('Hello & World'); // Returns: 'Hello &amp; World'
* text('5 < 10'); // Returns: '5 &lt; 10'
* text('Hello "World"'); // Returns: 'Hello "World"' (quotes OK in text)
*
* @see https://www.w3.org/TR/xml/#syntax
*/
export declare function text(txt: string): string;
/**
* Generates an opening XML tag with optional attributes.
*
* **Security Model:**
* - Validates attribute names to prevent injection via malformed names
* - Escapes all attribute values with proper XML entity encoding
* - Escapes `&`, `<`, `>`, `"`, and `'` in attribute values
* - Removes invalid XML Unicode characters
*
* Attribute values use full escaping (including quotes) because they appear
* within quoted strings in the XML output: `<tag attr="value">`.
*
* @param nodeName - The XML element name (e.g., 'url', 'loc', 'video:title')
* @param attrs - Optional object mapping attribute names to string values
* @param selfClose - If true, generates a self-closing tag (e.g., `<tag/>`)
* @returns Opening XML tag string
* @throws {InvalidXMLAttributeNameError} If an attribute name contains invalid characters
* @throws {TypeError} If nodeName is not a string or attrs values are not strings
*
* @example
* otag('url'); // Returns: '<url>'
* otag('video:player_loc', { autoplay: 'ap=1' }); // Returns: '<video:player_loc autoplay="ap=1">'
* otag('image:image', {}, true); // Returns: '<image:image/>'
*
* @see https://www.w3.org/TR/xml/#NT-Attribute
*/
export declare function otag(nodeName: TagNames | IndexTagNames, attrs?: StringObj, selfClose?: boolean): string;
/**
* Generates a closing XML tag.
*
* @param nodeName - The XML element name (e.g., 'url', 'loc', 'video:title')
* @returns Closing XML tag string
* @throws {TypeError} If nodeName is not a string
*
* @example
* ctag('url'); // Returns: '</url>'
* ctag('video:title'); // Returns: '</video:title>'
*/
export declare function ctag(nodeName: TagNames | IndexTagNames): string;
/**
* Generates a complete XML element with optional attributes and text content.
*
* This is a convenience function that combines `otag()`, `text()`, and `ctag()`.
* It supports three usage patterns via function overloading:
*
* 1. Element with text content: `element('loc', 'https://example.com')`
* 2. Element with attributes and text: `element('video:player_loc', { autoplay: 'ap=1' }, 'https://...')`
* 3. Self-closing element with attributes: `element('image:image', { href: '...' })`
*
* @param nodeName - The XML element name
* @param attrs - Either a string (text content) or object (attributes)
* @param innerText - Optional text content when attrs is an object
* @returns Complete XML element string
* @throws {InvalidXMLAttributeNameError} If an attribute name contains invalid characters
* @throws {TypeError} If arguments have invalid types
*
* @example
* // Pattern 1: Simple element with text
* element('loc', 'https://example.com')
* // Returns: '<loc>https://example.com</loc>'
*
* @example
* // Pattern 2: Element with attributes and text
* element('video:player_loc', { autoplay: 'ap=1' }, 'https://example.com/video')
* // Returns: '<video:player_loc autoplay="ap=1">https://example.com/video</video:player_loc>'
*
* @example
* // Pattern 3: Self-closing element with attributes
* element('xhtml:link', { rel: 'alternate', href: 'https://example.com/fr' })
* // Returns: '<xhtml:link rel="alternate" href="https://example.com/fr"/>'
*/
export declare function element(nodeName: TagNames, attrs: StringObj, innerText: string): string;
export declare function element(nodeName: TagNames | IndexTagNames, innerText: string): string;
export declare function element(nodeName: TagNames, attrs: StringObj): string;
+182
View File
@@ -0,0 +1,182 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
import { TagNames, IndexTagNames } from './types.js';
import { InvalidXMLAttributeNameError } from './errors.js';
/**
* Regular expression matching invalid XML 1.0 Unicode characters that must be removed.
*
* Based on the XML 1.0 specification (https://www.w3.org/TR/xml/#charsets):
* - Control characters (U+0000-U+001F except tab, newline, carriage return)
* - Delete character (U+007F)
* - Invalid control characters (U+0080-U+009F except U+0085)
* - Surrogate pairs (U+D800-U+DFFF)
* - Non-characters (\p{NChar} - permanently reserved code points)
*
* Performance note: This regex uses Unicode property escapes and may be slower
* on very large strings (100KB+). Consider pre-validation for untrusted input.
*
* @see https://www.w3.org/TR/xml/#charsets
*/
const invalidXMLUnicodeRegex =
// eslint-disable-next-line no-control-regex
/[\u0000-\u0008\u000B\u000C\u000E-\u001F\u007F-\u0084\u0086-\u009F\uD800-\uDFFF\p{NChar}]/gu;
/**
* Regular expressions for XML entity escaping
*/
const amp = /&/g;
const lt = /</g;
const gt = />/g;
const apos = /'/g;
const quot = /"/g;
/**
* Valid XML attribute name pattern. XML names must:
* - Start with a letter, underscore, or colon
* - Contain only letters, digits, hyphens, underscores, colons, or periods
*
* This is a simplified validation that accepts the most common attribute names.
* Note: In practice, this library only uses namespaced attributes like "video:title"
* which are guaranteed to be valid.
*
* @see https://www.w3.org/TR/xml/#NT-Name
*/
const validAttributeNameRegex = /^[a-zA-Z_:][\w:.-]*$/;
/**
* Validates that an attribute name is a valid XML identifier.
*
* XML attribute names must start with a letter, underscore, or colon,
* and contain only alphanumeric characters, hyphens, underscores, colons, or periods.
*
* @param name - The attribute name to validate
* @throws {InvalidXMLAttributeNameError} If the attribute name is invalid
*
* @example
* validateAttributeName('href'); // OK
* validateAttributeName('xml:lang'); // OK
* validateAttributeName('data-value'); // OK
* validateAttributeName('<script>'); // Throws InvalidXMLAttributeNameError
*/
function validateAttributeName(name) {
if (!validAttributeNameRegex.test(name)) {
throw new InvalidXMLAttributeNameError(name);
}
}
/**
* Escapes text content for safe inclusion in XML text nodes.
*
* **Security Model:**
* - Escapes `&` → `&amp;` (required to prevent entity interpretation)
* - Escapes `<` → `&lt;` (required to prevent tag injection)
* - Escapes `>` → `&gt;` (defense-in-depth, prevents CDATA injection)
* - Does NOT escape `"` or `'` (not required in text content, only in attributes)
* - Removes invalid XML Unicode characters per XML 1.0 spec
*
* **Why quotes aren't escaped:**
* In XML text content (between tags), quotes have no special meaning and don't
* need escaping. They only need escaping in attribute values, which is handled
* by the `otag()` function.
*
* @param txt - The text content to escape
* @returns XML-safe escaped text with invalid characters removed
* @throws {TypeError} If txt is not a string
*
* @example
* text('Hello & World'); // Returns: 'Hello &amp; World'
* text('5 < 10'); // Returns: '5 &lt; 10'
* text('Hello "World"'); // Returns: 'Hello "World"' (quotes OK in text)
*
* @see https://www.w3.org/TR/xml/#syntax
*/
export function text(txt) {
if (typeof txt !== 'string') {
throw new TypeError(`text() requires a string, received ${typeof txt}: ${String(txt)}`);
}
return txt
.replace(amp, '&amp;')
.replace(lt, '&lt;')
.replace(gt, '&gt;')
.replace(invalidXMLUnicodeRegex, '');
}
/**
* Generates an opening XML tag with optional attributes.
*
* **Security Model:**
* - Validates attribute names to prevent injection via malformed names
* - Escapes all attribute values with proper XML entity encoding
* - Escapes `&`, `<`, `>`, `"`, and `'` in attribute values
* - Removes invalid XML Unicode characters
*
* Attribute values use full escaping (including quotes) because they appear
* within quoted strings in the XML output: `<tag attr="value">`.
*
* @param nodeName - The XML element name (e.g., 'url', 'loc', 'video:title')
* @param attrs - Optional object mapping attribute names to string values
* @param selfClose - If true, generates a self-closing tag (e.g., `<tag/>`)
* @returns Opening XML tag string
* @throws {InvalidXMLAttributeNameError} If an attribute name contains invalid characters
* @throws {TypeError} If nodeName is not a string or attrs values are not strings
*
* @example
* otag('url'); // Returns: '<url>'
* otag('video:player_loc', { autoplay: 'ap=1' }); // Returns: '<video:player_loc autoplay="ap=1">'
* otag('image:image', {}, true); // Returns: '<image:image/>'
*
* @see https://www.w3.org/TR/xml/#NT-Attribute
*/
export function otag(nodeName, attrs, selfClose = false) {
if (typeof nodeName !== 'string') {
throw new TypeError(`otag() nodeName must be a string, received ${typeof nodeName}: ${String(nodeName)}`);
}
let attrstr = '';
for (const k in attrs) {
// Validate attribute name to prevent injection
validateAttributeName(k);
const attrValue = attrs[k];
if (typeof attrValue !== 'string') {
throw new TypeError(`otag() attribute "${k}" value must be a string, received ${typeof attrValue}: ${String(attrValue)}`);
}
// Escape attribute value with full entity encoding
const val = attrValue
.replace(amp, '&amp;')
.replace(lt, '&lt;')
.replace(gt, '&gt;')
.replace(apos, '&apos;')
.replace(quot, '&quot;')
.replace(invalidXMLUnicodeRegex, '');
attrstr += ` ${k}="${val}"`;
}
return `<${nodeName}${attrstr}${selfClose ? '/' : ''}>`;
}
/**
* Generates a closing XML tag.
*
* @param nodeName - The XML element name (e.g., 'url', 'loc', 'video:title')
* @returns Closing XML tag string
* @throws {TypeError} If nodeName is not a string
*
* @example
* ctag('url'); // Returns: '</url>'
* ctag('video:title'); // Returns: '</video:title>'
*/
export function ctag(nodeName) {
if (typeof nodeName !== 'string') {
throw new TypeError(`ctag() nodeName must be a string, received ${typeof nodeName}: ${String(nodeName)}`);
}
return `</${nodeName}>`;
}
export function element(nodeName, attrs, innerText) {
if (typeof attrs === 'string') {
// Pattern 1: element(nodeName, textContent)
return otag(nodeName) + text(attrs) + ctag(nodeName);
}
else if (innerText !== undefined) {
// Pattern 2: element(nodeName, attrs, textContent)
return otag(nodeName, attrs) + text(innerText) + ctag(nodeName);
}
else {
// Pattern 3: element(nodeName, attrs) - self-closing
return otag(nodeName, attrs, true);
}
}
+485
View File
@@ -0,0 +1,485 @@
import { URL } from 'node:url';
import type { TransformOptions } from 'node:stream';
/**
* How frequently the page is likely to change. This value provides general
* information to search engines and may not correlate exactly to how often they crawl the page. Please note that the
* value of this tag is considered a hint and not a command. See
* <https://www.sitemaps.org/protocol.html#xmlTagDefinitions> for the acceptable
* values
*/
export declare enum EnumChangefreq {
DAILY = "daily",
MONTHLY = "monthly",
ALWAYS = "always",
HOURLY = "hourly",
WEEKLY = "weekly",
YEARLY = "yearly",
NEVER = "never"
}
export declare enum EnumYesNo {
YES = "YES",
NO = "NO",
Yes = "Yes",
No = "No",
yes = "yes",
no = "no"
}
export declare enum EnumAllowDeny {
ALLOW = "allow",
DENY = "deny"
}
/**
* https://support.google.com/webmasters/answer/74288?hl=en&ref_topic=4581190
*/
export interface NewsItem {
access?: 'Registration' | 'Subscription';
publication: {
name: string;
/**
* The `<language>` is the language of your publication. Use an ISO 639
* language code (2 or 3 letters).
*/
language: string;
};
/**
* @example 'PressRelease, Blog'
*/
genres?: string;
/**
* Article publication date in W3C format, using either the "complete date" (YYYY-MM-DD) format or the "complete date
* plus hours, minutes, and seconds"
*/
publication_date: string;
/**
* The title of the news article
* @example 'Companies A, B in Merger Talks'
*/
title: string;
/**
* @example 'business, merger, acquisition'
*/
keywords?: string;
/**
* @example 'NASDAQ:A, NASDAQ:B'
*/
stock_tickers?: string;
}
/**
* Sitemap Image
* https://support.google.com/webmasters/answer/178636?hl=en&ref_topic=4581190
*/
export interface Img {
/**
* The URL of the image
* @example 'https://example.com/image.jpg'
*/
url: string;
/**
* The caption of the image
* @example 'Thanksgiving dinner'
*/
caption?: string;
/**
* The title of the image
* @example 'Star Wars EP IV'
*/
title?: string;
/**
* The geographic location of the image.
* @example 'Limerick, Ireland'
*/
geoLocation?: string;
/**
* A URL to the license of the image.
* @example 'https://example.com/license.txt'
*/
license?: string;
}
interface VideoItemBase {
/**
* A URL pointing to the video thumbnail image file
* @example "https://rtv3-img-roosterteeth.akamaized.net/store/0e841100-289b-4184-ae30-b6a16736960a.jpg/sm/thumb3.jpg"
*/
thumbnail_loc: string;
/**
* The title of the video
* @example '2018:E6 - GoldenEye: Source'
*/
title: string;
/**
* A description of the video. Maximum 2048 characters.
* @example 'We play gun game in GoldenEye: Source with a good friend of ours. His name is Gruchy. Dan Gruchy.'
*/
description: string;
/**
* A URL pointing to the actual video media file. Should be one of the supported formats. HTML is not a supported
* format. Flash is allowed, but no longer supported on most mobile platforms, and so may be indexed less well. Must
* not be the same as the `<loc>` URL.
* @example "http://streamserver.example.com/video123.mp4"
*/
content_loc?: string;
/**
* A URL pointing to a player for a specific video. Usually this is the information in the src element of an `<embed>`
* tag. Must not be the same as the `<loc>` URL
* @example "https://roosterteeth.com/embed/rouletsplay-2018-goldeneye-source"
*/
player_loc?: string;
/**
* A string the search engine can append as a query param to enable automatic
* playback. Equivilant to auto play attr on player_loc tag.
* @example 'ap=1'
*/
'player_loc:autoplay'?: string;
/**
* Whether the search engine can embed the video in search results. Allowed values are yes or no.
*/
'player_loc:allow_embed'?: EnumYesNo;
/**
* The length of the video in seconds
* @example 600
*/
duration?: number;
/**
* The date after which the video will no longer be available.
* @example "2012-07-16T19:20:30+08:00"
*/
expiration_date?: string;
/**
* The number of times the video has been viewed
*/
view_count?: number;
/**
* The date the video was first published, in W3C format.
* @example "2012-07-16T19:20:30+08:00"
*/
publication_date?: string;
/**
* A short description of the broad category that the video belongs to. This is a string no longer than 256 characters.
* @example Baking
*/
category?: string;
/**
* Whether to show or hide your video in search results from specific countries.
* @example "IE GB US CA"
*/
restriction?: string;
/**
* Whether the countries in restriction are allowed or denied
* @example 'deny'
*/
'restriction:relationship'?: EnumAllowDeny;
gallery_loc?: string;
/**
* [Optional] Specifies the URL of a webpage with additional information about this uploader. This URL must be in the same domain as the <loc> tag.
* @see https://developers.google.com/search/docs/advanced/sitemaps/video-sitemaps
* @example http://www.example.com/users/grillymcgrillerson
*/
'uploader:info'?: string;
'gallery_loc:title'?: string;
/**
* The price to download or view the video. Omit this tag for free videos.
* @example "1.99"
*/
price?: string;
/**
* Specifies the resolution of the purchased version. Supported values are hd and sd.
* @example "HD"
*/
'price:resolution'?: Resolution;
/**
* Specifies the currency in ISO4217 format.
* @example "USD"
*/
'price:currency'?: string;
/**
* Specifies the purchase option. Supported values are rend and own.
* @example "rent"
*/
'price:type'?: PriceType;
/**
* The video uploader's name. Only one <video:uploader> is allowed per video. String value, max 255 characters.
* @example "GrillyMcGrillerson"
*/
uploader?: string;
/**
* Whether to show or hide your video in search results on specified platform types. This is a list of space-delimited
* platform types. See <https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190> for more detail
* @example "tv"
*/
platform?: string;
id?: string;
'platform:relationship'?: EnumAllowDeny;
}
/**
* Video price type - supports both lowercase and uppercase variants
* as allowed by the Google Video Sitemap specification
* @see https://developers.google.com/search/docs/advanced/sitemaps/video-sitemaps
*/
export type PriceType = 'rent' | 'purchase' | 'RENT' | 'PURCHASE';
/**
* Video resolution - supports both lowercase and uppercase variants
* as allowed by the Google Video Sitemap specification
* @see https://developers.google.com/search/docs/advanced/sitemaps/video-sitemaps
*/
export type Resolution = 'HD' | 'hd' | 'sd' | 'SD';
/**
* Sitemap video. <https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190>
*/
export interface VideoItem extends VideoItemBase {
/**
* An arbitrary string tag describing the video. Tags are generally very short descriptions of key concepts associated
* with a video or piece of content.
* @example ['Baking']
*/
tag: string[];
/**
* The rating of the video. Supported values are float numbers.
* @example 2.5
*/
rating?: number;
family_friendly?: EnumYesNo;
/**
* Indicates whether a subscription (either paid or free) is required to view
* the video. Allowed values are yes or no.
*/
requires_subscription?: EnumYesNo;
/**
* Indicates whether the video is a live stream. Supported values are yes or no.
*/
live?: EnumYesNo;
}
/**
* Sitemap video. <https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190>
*/
export interface VideoItemLoose extends VideoItemBase {
/**
* An arbitrary string tag describing the video. Tags are generally very short descriptions of key concepts associated
* with a video or piece of content.
* @example ['Baking']
*/
tag?: string | string[];
/**
* The rating of the video. Supported values are float numbers.
* @example 2.5
*/
rating?: string | number;
family_friendly?: EnumYesNo | boolean;
requires_subscription?: EnumYesNo | boolean;
/**
* Indicates whether the video is a live stream. Supported values are yes or no.
*/
live?: EnumYesNo | boolean;
}
/**
* https://support.google.com/webmasters/answer/189077
*/
export interface LinkItem {
/**
* @example 'en'
*/
lang: string;
/**
* @example 'en-us'
*/
hreflang?: string;
url: string;
}
export interface IndexItem {
url: string;
lastmod?: string;
}
interface SitemapItemBase {
lastmod?: string;
changefreq?: EnumChangefreq;
fullPrecisionPriority?: boolean;
news?: NewsItem;
expires?: string;
androidLink?: string;
ampLink?: string;
url: string;
}
/**
* Strict options for individual sitemap entries
*/
export interface SitemapItem extends SitemapItemBase {
priority?: number;
img: Img[];
video: VideoItem[];
links: LinkItem[];
}
/**
* Options for individual sitemap entries prior to normalization
*/
export interface SitemapItemLoose extends SitemapItemBase {
/**
* A number between 0 and 1. A numeric string such as `'0.5'` is converted
* to a number.
*/
priority?: number | string;
video?: VideoItemLoose | VideoItemLoose[];
img?: string | Img | (string | Img)[];
links?: LinkItem[];
lastmodfile?: string | Buffer | URL;
lastmodISO?: string;
lastmodrealtime?: boolean;
}
/**
* How to handle errors in passed in urls
*/
export declare enum ErrorLevel {
/**
* Validation will be skipped and nothing logged or thrown.
*/
SILENT = "silent",
/**
* If an invalid value is encountered, the stream's `logger` is called with
* details (console.warn by default)
*/
WARN = "warn",
/**
* An Error will be thrown on encountering invalid data.
*/
THROW = "throw"
}
export type ErrorHandler = (error: Error, level: ErrorLevel) => void;
/**
* Receives every message the library would otherwise print to the console.
* Pass one as the `logger` option of any stream, or pass `false` to silence it.
*/
export type Logger = (level: 'warn' | 'error' | 'info' | 'log', ...message: Parameters<Console['log']>) => void;
export declare enum TagNames {
url = "url",
loc = "loc",
urlset = "urlset",
lastmod = "lastmod",
changefreq = "changefreq",
priority = "priority",
'video:thumbnail_loc' = "video:thumbnail_loc",
'video:video' = "video:video",
'video:title' = "video:title",
'video:description' = "video:description",
'video:tag' = "video:tag",
'video:duration' = "video:duration",
'video:player_loc' = "video:player_loc",
'video:content_loc' = "video:content_loc",
'image:image' = "image:image",
'image:loc' = "image:loc",
'image:geo_location' = "image:geo_location",
'image:license' = "image:license",
'image:title' = "image:title",
'image:caption' = "image:caption",
'video:requires_subscription' = "video:requires_subscription",
'video:publication_date' = "video:publication_date",
'video:id' = "video:id",
'video:restriction' = "video:restriction",
'video:family_friendly' = "video:family_friendly",
'video:view_count' = "video:view_count",
'video:uploader' = "video:uploader",
'video:expiration_date' = "video:expiration_date",
'video:platform' = "video:platform",
'video:price' = "video:price",
'video:rating' = "video:rating",
'video:category' = "video:category",
'video:live' = "video:live",
'video:gallery_loc' = "video:gallery_loc",
'news:news' = "news:news",
'news:publication' = "news:publication",
'news:name' = "news:name",
'news:access' = "news:access",
'news:genres' = "news:genres",
'news:publication_date' = "news:publication_date",
'news:title' = "news:title",
'news:keywords' = "news:keywords",
'news:stock_tickers' = "news:stock_tickers",
'news:language' = "news:language",
'mobile:mobile' = "mobile:mobile",
'xhtml:link' = "xhtml:link",
expires = "expires"
}
export declare enum IndexTagNames {
sitemap = "sitemap",
sitemapindex = "sitemapindex",
loc = "loc",
lastmod = "lastmod"
}
/**
* Generic object with string keys and any values
* Used for XML attribute building and other flexible data structures
*/
export interface StringObj {
[index: string]: any;
}
/**
* Receives the joined character data for an element, or a message when the
* element's character data was dropped for exceeding the buffer cap.
* Internal: index.ts does not re-export it.
*/
export interface CharDataHandlers {
onText: (tag: string, text: string) => void;
onOverflow: (message: string) => void;
}
/**
* Buffers a SAX parser's character data (text and CDATA) per element and
* hands it to CharDataHandlers at each tag boundary.
* Internal: index.ts does not re-export it.
*/
export interface CharDataBuffer {
openTag: (name: string) => void;
append: (text: string) => void;
closeTag: () => void;
}
/**
* Options shared by the XML parser streams (`XMLToSitemapItemStream` and
* `XMLToSitemapIndexStream`).
*/
export interface SaxParserStreamOptions extends TransformOptions {
level?: ErrorLevel;
logger?: Logger | false;
/**
* Maximum number of entries (`<url>` or `<sitemap>` elements) to emit.
* Entries past the cap are dropped and reported once as an error, so
* untrusted input can't grow memory without bound. A non-negative integer,
* or `Infinity` for no cap.
* @default 50000 (per sitemaps.org)
*/
maxEntries?: number;
}
/** Options for `XMLToSitemapItemStream` */
export interface XMLToSitemapItemStreamOptions extends SaxParserStreamOptions {
}
/** Options for `XMLToSitemapIndexStream` */
export interface XMLToSitemapIndexItemStreamOptions extends SaxParserStreamOptions {
}
/**
* What a parser subclass tells the shared `SaxParserStream` base. Internal:
* index.ts does not re-export it.
*/
export interface SaxParserStreamConfig {
/** Whether a tag belongs to the parsed format; others are reported */
isKnownTag: (name: string) => boolean;
/** Error text for the first entry past the entry cap */
overflowMessage: (maxEntries: number) => string;
}
/**
* Options for `openSitemapFile`, which writes one sitemap of a
* `SitemapAndIndexStream` to `sitemap-N.xml` (or `sitemap-N.xml.gz`).
* Internal: shared by `simpleSitemapAndIndex` and the CLI; index.ts does not
* re-export it.
*/
export interface SitemapFileOptions {
/** Directory the sitemap file is written to */
destinationDir: string;
/** Gzip the file and name it `.xml.gz` */
gzip: boolean;
/** Builds the sitemap's public URL from its file name, such as `sitemap-0.xml.gz` */
toURL: (fileName: string) => string;
/** Names where the URL came from in a validation error */
urlParamName: string;
/**
* Called with any error from the sitemap-to-file pipeline. Usually destroys
* the `SitemapAndIndexStream`, which also covers failures after it has
* stopped watching this file.
*/
onError: (err: Error) => void;
}
export {};
+108
View File
@@ -0,0 +1,108 @@
import { URL } from 'node:url';
/**
* How frequently the page is likely to change. This value provides general
* information to search engines and may not correlate exactly to how often they crawl the page. Please note that the
* value of this tag is considered a hint and not a command. See
* <https://www.sitemaps.org/protocol.html#xmlTagDefinitions> for the acceptable
* values
*/
export var EnumChangefreq;
(function (EnumChangefreq) {
EnumChangefreq["DAILY"] = "daily";
EnumChangefreq["MONTHLY"] = "monthly";
EnumChangefreq["ALWAYS"] = "always";
EnumChangefreq["HOURLY"] = "hourly";
EnumChangefreq["WEEKLY"] = "weekly";
EnumChangefreq["YEARLY"] = "yearly";
EnumChangefreq["NEVER"] = "never";
})(EnumChangefreq || (EnumChangefreq = {}));
export var EnumYesNo;
(function (EnumYesNo) {
EnumYesNo["YES"] = "YES";
EnumYesNo["NO"] = "NO";
EnumYesNo["Yes"] = "Yes";
EnumYesNo["No"] = "No";
EnumYesNo["yes"] = "yes";
EnumYesNo["no"] = "no";
})(EnumYesNo || (EnumYesNo = {}));
export var EnumAllowDeny;
(function (EnumAllowDeny) {
EnumAllowDeny["ALLOW"] = "allow";
EnumAllowDeny["DENY"] = "deny";
})(EnumAllowDeny || (EnumAllowDeny = {}));
/**
* How to handle errors in passed in urls
*/
export var ErrorLevel;
(function (ErrorLevel) {
/**
* Validation will be skipped and nothing logged or thrown.
*/
ErrorLevel["SILENT"] = "silent";
/**
* If an invalid value is encountered, the stream's `logger` is called with
* details (console.warn by default)
*/
ErrorLevel["WARN"] = "warn";
/**
* An Error will be thrown on encountering invalid data.
*/
ErrorLevel["THROW"] = "throw";
})(ErrorLevel || (ErrorLevel = {}));
export var TagNames;
(function (TagNames) {
TagNames["url"] = "url";
TagNames["loc"] = "loc";
TagNames["urlset"] = "urlset";
TagNames["lastmod"] = "lastmod";
TagNames["changefreq"] = "changefreq";
TagNames["priority"] = "priority";
TagNames["video:thumbnail_loc"] = "video:thumbnail_loc";
TagNames["video:video"] = "video:video";
TagNames["video:title"] = "video:title";
TagNames["video:description"] = "video:description";
TagNames["video:tag"] = "video:tag";
TagNames["video:duration"] = "video:duration";
TagNames["video:player_loc"] = "video:player_loc";
TagNames["video:content_loc"] = "video:content_loc";
TagNames["image:image"] = "image:image";
TagNames["image:loc"] = "image:loc";
TagNames["image:geo_location"] = "image:geo_location";
TagNames["image:license"] = "image:license";
TagNames["image:title"] = "image:title";
TagNames["image:caption"] = "image:caption";
TagNames["video:requires_subscription"] = "video:requires_subscription";
TagNames["video:publication_date"] = "video:publication_date";
TagNames["video:id"] = "video:id";
TagNames["video:restriction"] = "video:restriction";
TagNames["video:family_friendly"] = "video:family_friendly";
TagNames["video:view_count"] = "video:view_count";
TagNames["video:uploader"] = "video:uploader";
TagNames["video:expiration_date"] = "video:expiration_date";
TagNames["video:platform"] = "video:platform";
TagNames["video:price"] = "video:price";
TagNames["video:rating"] = "video:rating";
TagNames["video:category"] = "video:category";
TagNames["video:live"] = "video:live";
TagNames["video:gallery_loc"] = "video:gallery_loc";
TagNames["news:news"] = "news:news";
TagNames["news:publication"] = "news:publication";
TagNames["news:name"] = "news:name";
TagNames["news:access"] = "news:access";
TagNames["news:genres"] = "news:genres";
TagNames["news:publication_date"] = "news:publication_date";
TagNames["news:title"] = "news:title";
TagNames["news:keywords"] = "news:keywords";
TagNames["news:stock_tickers"] = "news:stock_tickers";
TagNames["news:language"] = "news:language";
TagNames["mobile:mobile"] = "mobile:mobile";
TagNames["xhtml:link"] = "xhtml:link";
TagNames["expires"] = "expires";
})(TagNames || (TagNames = {}));
export var IndexTagNames;
(function (IndexTagNames) {
IndexTagNames["sitemap"] = "sitemap";
IndexTagNames["sitemapindex"] = "sitemapindex";
IndexTagNames["loc"] = "loc";
IndexTagNames["lastmod"] = "lastmod";
})(IndexTagNames || (IndexTagNames = {}));
+86
View File
@@ -0,0 +1,86 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
import { type WriteStream } from 'node:fs';
import { Readable, type ReadableOptions, type TransformOptions, type Writable } from 'node:stream';
import type { SitemapItem, SitemapItemLoose, CharDataBuffer, CharDataHandlers, SitemapFileOptions } from './types.js';
/**
* Per ADR 0001, CDATA sections are ordinary character data: wire both the SAX
* `text` and `cdata` events to `append`. Character data is joined, trimmed and
* handed to `onText` once at the next tag boundary, attributed to the innermost
* open element (which is not the last opened tag once a child has closed).
* An element whose character data exceeds LIMITS.MAX_PARSER_CHAR_DATA_LENGTH is
* dropped and reported through `onOverflow` to bound memory use.
*/
export declare function createCharDataBuffer({ onText, onOverflow, }: CharDataHandlers): CharDataBuffer;
/**
* Combines multiple streams into one
* @param streams the streams to combine
*/
export declare function mergeStreams(streams: Readable[], options?: TransformOptions): Readable;
export interface ReadlineStreamOptions extends ReadableOptions {
input: Readable;
}
/**
* Wraps node's ReadLine in a stream
*/
export declare class ReadlineStream extends Readable {
private _source;
constructor(options: ReadlineStreamOptions);
_read(size: number): void;
}
/**
* Takes a stream likely from fs.createReadStream('./path') and returns a stream
* of sitemap items
* @param stream a stream of line separated urls.
* @param opts.isJSON is the stream line separated JSON. leave undefined to guess
*/
export declare function lineSeparatedURLsToSitemapOptions(stream: Readable, { isJSON }?: {
isJSON?: boolean;
}): Readable;
/**
* Returns `err` if it is an Error, or wraps it in one, for passing a caught
* value to a stream callback
*/
export declare function toError(err: unknown): Error;
/**
* Converts the passed in sitemap entry into one capable of being consumed by SitemapItem
* @param {string | SitemapItemLoose} elem the string or object to be converted
* @param {string} hostname
* @returns SitemapItemOptions a strict sitemap item option
*/
export declare function normalizeURL(elem: string | SitemapItemLoose, hostname?: string, lastmodDateOnly?: boolean): SitemapItem;
/**
* Connects a sitemap to the stream that writes its file, through gzip when
* `gzip` is true. Uses `pipeline()`, which destroys every stream when one
* fails, so a gzip failure also fails `writeStream`, which
* `SitemapAndIndexStream` watches.
*
* @param onError - Called with the error if the pipeline fails.
*/
export declare function pipeSitemapToFile(sitemap: Readable, writeStream: Writable, gzip: boolean, onError: (err: Error) => void): void;
/**
* Walks up from `fromDir` to the filesystem root and returns the first path
* where `relativePath` exists, or undefined if there is none.
*
* Used to find files that ship with the package (`package.json`, the schema)
* from a module's own directory, whatever the build's output layout.
*/
export declare function findUp(relativePath: string, fromDir: string): string | undefined;
/**
* Opens sitemap number `index` as a file, for a `SitemapAndIndexStream`
* `getSitemapStream` callback: names it `sitemap-N.xml` (`.xml.gz` when
* gzipped), builds its URL, checks the URL before creating the file so a URL
* the index would reject leaves no empty file behind, and pipes `sitemap` into
* it.
*
* @returns the `[url, sitemap, writeStream]` tuple `getSitemapStream` returns
*/
export declare function openSitemapFile<S extends Readable>(index: number, sitemap: S, { destinationDir, gzip, toURL, urlParamName, onError }: SitemapFileOptions): [string, S, WriteStream];
/**
* Message text of a caught value, for embedding in a wrapping error's message.
* @param err - anything caught by a `catch` clause
*/
export declare function errorMessage(err: unknown): string;
+330
View File
@@ -0,0 +1,330 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
import { createWriteStream, existsSync, statSync, } from 'node:fs';
import { dirname, resolve } from 'node:path';
import { pipeline, Readable, Transform, PassThrough, } from 'node:stream';
import { createInterface, Interface } from 'node:readline';
import { createGzip } from 'node:zlib';
import { URL } from 'node:url';
import { EnumYesNo } from './types.js';
import { LIMITS } from './constants.js';
import { validateURL } from './validation.js';
/**
* Per ADR 0001, CDATA sections are ordinary character data: wire both the SAX
* `text` and `cdata` events to `append`. Character data is joined, trimmed and
* handed to `onText` once at the next tag boundary, attributed to the innermost
* open element (which is not the last opened tag once a child has closed).
* An element whose character data exceeds LIMITS.MAX_PARSER_CHAR_DATA_LENGTH is
* dropped and reported through `onOverflow` to bound memory use.
*/
export function createCharDataBuffer({ onText, onOverflow, }) {
let charData = '';
let overflow = false;
const openTags = [];
const flush = () => {
const tag = openTags[openTags.length - 1];
const text = charData.trim();
charData = '';
if (overflow) {
overflow = false;
onOverflow(`character data for ${tag} exceeds max length of ${LIMITS.MAX_PARSER_CHAR_DATA_LENGTH}`);
}
else if (text) {
onText(tag, text);
}
};
return {
openTag(name) {
flush();
openTags.push(name);
},
append(text) {
if (overflow) {
return;
}
if (charData.length + text.length > LIMITS.MAX_PARSER_CHAR_DATA_LENGTH) {
overflow = true;
charData = '';
return;
}
charData += text;
},
closeTag() {
flush();
openTags.pop();
},
};
}
/**
* Combines multiple streams into one
* @param streams the streams to combine
*/
export function mergeStreams(streams, options) {
let pass = new PassThrough(options);
let waiting = streams.length;
for (const stream of streams) {
pass = stream.pipe(pass, { end: false });
stream.once('end', () => --waiting === 0 && pass.emit('end'));
}
return pass;
}
/**
* Wraps node's ReadLine in a stream
*/
export class ReadlineStream extends Readable {
_source;
constructor(options) {
if (options.autoDestroy === undefined) {
options.autoDestroy = true;
}
options.objectMode = true;
super(options);
this._source = createInterface({
input: options.input,
terminal: false,
crlfDelay: Infinity,
});
// Every time there's data, push it into the internal buffer.
this._source.on('line', (chunk) => {
// If push() returns false, then stop reading from source.
if (!this.push(chunk))
this._source.pause();
});
// When the source ends, push the EOF-signaling `null` chunk.
this._source.on('close', () => {
this.push(null);
});
}
// _read() will be called when the stream wants to pull more data in.
// The advisory size argument is ignored in this case.
_read(size) {
this._source.resume();
}
}
/**
* Takes a stream likely from fs.createReadStream('./path') and returns a stream
* of sitemap items
* @param stream a stream of line separated urls.
* @param opts.isJSON is the stream line separated JSON. leave undefined to guess
*/
export function lineSeparatedURLsToSitemapOptions(stream, { isJSON } = {}) {
return new ReadlineStream({ input: stream }).pipe(new Transform({
objectMode: true,
transform: (line, encoding, cb) => {
if (isJSON || (isJSON === undefined && line[0] === '{')) {
cb(null, JSON.parse(line));
}
else {
cb(null, line);
}
},
}));
}
/**
* Returns `err` if it is an Error, or wraps it in one, for passing a caught
* value to a stream callback
*/
export function toError(err) {
return err instanceof Error ? err : new Error(String(err));
}
function boolToYESNO(bool) {
if (bool === undefined) {
return undefined;
}
if (typeof bool === 'boolean') {
return bool ? EnumYesNo.yes : EnumYesNo.no;
}
return bool;
}
/**
* Converts a numeric string priority, such as `'0.5'`, to a number. Returns
* any other value unchanged, for validateSMIOptions to report.
*/
function toPriority(priority) {
if (typeof priority === 'string' &&
LIMITS.PRIORITY_STRING_REGEX.test(priority)) {
return Number(priority);
}
// Not a number: passed through so validation reports it at WARN and THROW
return priority;
}
/**
* Converts the passed in sitemap entry into one capable of being consumed by SitemapItem
* @param {string | SitemapItemLoose} elem the string or object to be converted
* @param {string} hostname
* @returns SitemapItemOptions a strict sitemap item option
*/
export function normalizeURL(elem, hostname, lastmodDateOnly = false) {
// SitemapItem
// create object with url property
const smi = {
img: [],
video: [],
links: [],
url: '',
};
if (typeof elem === 'string') {
smi.url = new URL(elem, hostname).toString();
return smi;
}
const { url, img, links, video, lastmodfile, lastmodISO, lastmod, priority, ...other } = elem;
Object.assign(smi, other);
if (priority !== undefined) {
smi.priority = toPriority(priority);
}
smi.url = new URL(url, hostname).toString();
if (img) {
// prepend hostname to all image urls
smi.img = (Array.isArray(img) ? img : [img]).map((el) => typeof el === 'string'
? { url: new URL(el, hostname).toString() }
: { ...el, url: new URL(el.url, hostname).toString() });
}
if (links) {
smi.links = links.map((link) => ({
...link,
url: new URL(link.url, hostname).toString(),
}));
}
if (video) {
smi.video = (Array.isArray(video) ? video : [video]).map((video) => {
const nv = {
...video,
family_friendly: boolToYESNO(video.family_friendly),
live: boolToYESNO(video.live),
requires_subscription: boolToYESNO(video.requires_subscription),
tag: [],
rating: undefined,
};
if (video.tag !== undefined) {
nv.tag = !Array.isArray(video.tag) ? [video.tag] : video.tag;
}
if (video.rating !== undefined) {
if (typeof video.rating === 'string') {
const parsedRating = parseFloat(video.rating);
// Validate parsed rating is a valid number
if (Number.isNaN(parsedRating)) {
throw new Error(`Invalid video rating "${video.rating}" for URL "${elem.url}": must be a valid number`);
}
nv.rating = parsedRating;
}
else {
nv.rating = video.rating;
}
}
if (typeof video.view_count === 'string') {
const parsedViewCount = parseInt(video.view_count, 10);
// Validate parsed view count is a valid non-negative integer
if (Number.isNaN(parsedViewCount)) {
throw new Error(`Invalid video view_count "${video.view_count}" for URL "${elem.url}": must be a valid number`);
}
if (parsedViewCount < 0) {
throw new Error(`Invalid video view_count "${video.view_count}" for URL "${elem.url}": cannot be negative`);
}
nv.view_count = parsedViewCount;
}
else if (typeof video.view_count === 'number') {
nv.view_count = video.view_count;
}
return nv;
});
}
// If given a file to use for last modified date
if (lastmodfile) {
const { mtime } = statSync(lastmodfile);
const lastmodDate = new Date(mtime);
// Validate date is valid
if (Number.isNaN(lastmodDate.getTime())) {
throw new Error(`Invalid date from file stats for URL "${smi.url}": file modification time is invalid`);
}
smi.lastmod = lastmodDate.toISOString();
// The date of last modification (YYYY-MM-DD)
}
else if (lastmodISO) {
const lastmodDate = new Date(lastmodISO);
// Validate date is valid
if (Number.isNaN(lastmodDate.getTime())) {
throw new Error(`Invalid lastmodISO "${lastmodISO}" for URL "${smi.url}": must be a valid date string`);
}
smi.lastmod = lastmodDate.toISOString();
}
else if (lastmod) {
const lastmodDate = new Date(lastmod);
// Validate date is valid
if (Number.isNaN(lastmodDate.getTime())) {
throw new Error(`Invalid lastmod "${lastmod}" for URL "${smi.url}": must be a valid date string`);
}
smi.lastmod = lastmodDate.toISOString();
}
if (lastmodDateOnly && smi.lastmod) {
smi.lastmod = smi.lastmod.slice(0, 10);
}
return smi;
}
/**
* Connects a sitemap to the stream that writes its file, through gzip when
* `gzip` is true. Uses `pipeline()`, which destroys every stream when one
* fails, so a gzip failure also fails `writeStream`, which
* `SitemapAndIndexStream` watches.
*
* @param onError - Called with the error if the pipeline fails.
*/
export function pipeSitemapToFile(sitemap, writeStream, gzip, onError) {
const streams = gzip
? [sitemap, createGzip(), writeStream]
: [sitemap, writeStream];
pipeline(streams, (err) => {
if (err) {
onError(err);
}
});
}
/**
* Walks up from `fromDir` to the filesystem root and returns the first path
* where `relativePath` exists, or undefined if there is none.
*
* Used to find files that ship with the package (`package.json`, the schema)
* from a module's own directory, whatever the build's output layout.
*/
export function findUp(relativePath, fromDir) {
let dir = fromDir;
for (;;) {
const candidate = resolve(dir, relativePath);
if (existsSync(candidate)) {
return candidate;
}
const parent = dirname(dir);
if (parent === dir) {
return undefined;
}
dir = parent;
}
}
/**
* Opens sitemap number `index` as a file, for a `SitemapAndIndexStream`
* `getSitemapStream` callback: names it `sitemap-N.xml` (`.xml.gz` when
* gzipped), builds its URL, checks the URL before creating the file so a URL
* the index would reject leaves no empty file behind, and pipes `sitemap` into
* it.
*
* @returns the `[url, sitemap, writeStream]` tuple `getSitemapStream` returns
*/
export function openSitemapFile(index, sitemap, { destinationDir, gzip, toURL, urlParamName, onError }) {
const fileName = `sitemap-${index}.xml${gzip ? '.gz' : ''}`;
const url = toURL(fileName);
validateURL(url, urlParamName);
const writeStream = createWriteStream(resolve(destinationDir, fileName));
// SitemapAndIndexStream watches writeStream; onError reports any failure
// that comes after it stopped watching.
pipeSitemapToFile(sitemap, writeStream, gzip, onError);
return [url, sitemap, writeStream];
}
/**
* Message text of a caught value, for embedding in a wrapping error's message.
* @param err - anything caught by a `catch` clause
*/
export function errorMessage(err) {
return err instanceof Error ? err.message : String(err);
}
+136
View File
@@ -0,0 +1,136 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
import { ErrorLevel, EnumChangefreq, EnumYesNo, EnumAllowDeny } from './types.js';
import type { SitemapItem, PriceType, Resolution, ErrorHandler, LinkItem } from './types.js';
export declare const validators: {
[index: string]: RegExp;
};
/**
* Type guard to check if a string is a valid price type
*/
export declare function isPriceType(pt: string | PriceType): pt is PriceType;
/**
* Type guard to check if a string is a valid resolution
*/
export declare function isResolution(res: string): res is Resolution;
export declare function isValidChangeFreq(freq: string): freq is EnumChangefreq;
/**
* Type guard to check if a string is a valid yes/no value
*/
export declare function isValidYesNo(yn: string): yn is EnumYesNo;
/**
* Checks that a priority is a finite number within sitemaps.org bounds (0–1)
*/
export declare function isValidPriority(priority: unknown): boolean;
/**
* Checks that a video duration is a finite number of seconds within Google's
* video sitemap bounds (0–28800)
*/
export declare function isValidVideoDuration(duration: number): boolean;
/**
* Checks that a video rating is a finite number within Google's video sitemap
* bounds (0–5)
*/
export declare function isValidVideoRating(rating: number): boolean;
/**
* Checks that a video view count is a finite, non-negative number
*/
export declare function isValidVideoViewCount(count: number): boolean;
/**
* Checks that a date string is in ISO 8601 / W3C format
*/
export declare function isValidISODate(date: string): boolean;
/**
* Returns an alternate link's language (`lang`, else `hreflang`), or
* `undefined` when neither is a non-empty string. Validation reports such a
* link, and the writer drops it.
*/
export declare function getLinkHreflang(link: LinkItem): string | undefined;
/**
* Type guard to check if a string is a valid allow/deny value
*/
export declare function isAllowDeny(ad: string): ad is EnumAllowDeny;
/**
* Validates that a URL is well-formed and meets security requirements
*
* Security: This function enforces that URLs use safe protocols (http/https),
* are within reasonable length limits (2048 chars per sitemaps.org spec),
* and can be properly parsed. This prevents protocol injection attacks and
* ensures compliance with sitemap specifications.
*
* @param url - The URL to validate
* @param paramName - The parameter name for error messages
* @throws {InvalidURLError} If the URL is invalid (a subclass of InvalidHostnameError)
*/
export declare function validateURL(url: string, paramName: string): void;
/**
* Runs validateURL and returns its error message instead of throwing, for
* parsers that report invalid values and keep going.
*
* @returns The validation error message, or undefined if the URL is valid
*/
export declare function getURLError(url: string, paramName: string): string | undefined;
/**
* Validates that a path doesn't contain path traversal sequences
*
* Security: This function prevents path traversal attacks by detecting
* any occurrence of '..' in the path, whether it appears as '../', '/..',
* or standalone. This prevents attackers from accessing files outside
* the intended directory structure.
*
* @param path - The path to validate
* @param paramName - The parameter name for error messages
* @throws {InvalidPathError} If the path contains traversal sequences
*/
export declare function validatePath(path: string, paramName: string): void;
/**
* Validates that a public base path is safe for URL construction
*
* Security: This function prevents path traversal attacks and validates
* that the path is safe for use in URL construction within sitemap indexes.
* It checks for '..' sequences, null bytes, and invalid whitespace that
* could be used to manipulate URL structure or inject malicious content.
*
* @param publicBasePath - The public base path to validate
* @throws {InvalidPublicBasePathError} If the path is invalid
*/
export declare function validatePublicBasePath(publicBasePath: string): void;
/**
* Validates that a limit is within acceptable range per sitemaps.org spec
*
* Security: This function enforces sitemap size limits (1-50,000 URLs per
* sitemap) as specified by sitemaps.org. This prevents resource exhaustion
* attacks and ensures compliance with search engine requirements.
*
* @param limit - The limit to validate
* @throws {InvalidLimitError} If the limit is out of range
*/
export declare function validateLimit(limit: number): void;
/**
* Whether `n` is a valid parser entry cap: a non-negative integer, or
* `Infinity` for no cap. Unlike validateLimit, 0 and values above the
* sitemaps.org limit are allowed.
*/
export declare function isValidMaxEntries(n: number): boolean;
/**
* Validates that an XSL URL is safe and well-formed
*
* Security: This function validates XSL stylesheet URLs to prevent
* injection attacks. It blocks dangerous protocols and content patterns
* that could be used for XSS or other attacks. The validation uses
* case-insensitive matching to catch obfuscated attacks.
*
* @param xslUrl - The XSL URL to validate
* @throws {InvalidXSLUrlError} If the URL is invalid
*/
export declare function validateXSLUrl(xslUrl: string): void;
/**
* Verifies all data passed in will comply with sitemap spec.
* @param conf Options to validate
* @param level logging level
* @param errorHandler error handling func
*/
export declare function validateSMIOptions(conf: SitemapItem, level?: ErrorLevel, errorHandler?: ErrorHandler): SitemapItem;
+460
View File
@@ -0,0 +1,460 @@
/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
import { InvalidPathError, InvalidURLError, InvalidLimitError, InvalidPublicBasePathError, InvalidXSLUrlError, ChangeFreqInvalidError, InvalidAttrValue, InvalidNewsAccessValue, InvalidNewsFormat, InvalidVideoDescription, InvalidVideoDuration, InvalidVideoFormat, InvalidVideoRating, NoURLError, NoConfigError, PriorityInvalidError, InvalidVideoTitle, InvalidVideoViewCount, InvalidVideoTagCount, InvalidVideoCategory, InvalidVideoFamilyFriendly, InvalidVideoYesNo, InvalidLinkHreflang, InvalidVideoRestriction, InvalidVideoRestrictionRelationship, InvalidVideoPriceType, InvalidVideoResolution, InvalidVideoPriceCurrency, } from './errors.js';
import { ErrorLevel, EnumChangefreq, EnumYesNo, EnumAllowDeny, } from './types.js';
import { LIMITS } from './constants.js';
import { defaultLogger, loggingErrorHandler } from './logger.js';
import { isAbsolute } from 'node:path';
/**
* Validator regular expressions for various sitemap fields
*/
const allowDeny = /^(?:allow|deny)$/;
export const validators = {
'price:currency': /^[A-Z]{3}$/,
'price:type': /^(?:rent|purchase|RENT|PURCHASE)$/,
'price:resolution': /^(?:HD|hd|sd|SD)$/,
'platform:relationship': allowDeny,
'restriction:relationship': allowDeny,
restriction: /^([A-Z]{2}( +[A-Z]{2})*)?$/,
platform: /^((web|mobile|tv)( (web|mobile|tv))*)?$/,
// Language codes: zh-cn, zh-tw, or ISO 639 2-3 letter codes
language: /^(zh-cn|zh-tw|[a-z]{2,3})$/,
genres: /^(PressRelease|Satire|Blog|OpEd|Opinion|UserGenerated)(, *(PressRelease|Satire|Blog|OpEd|Opinion|UserGenerated))*$/,
stock_tickers: /^(\w+:\w+(, *\w+:\w+){0,4})?$/,
};
/**
* Type guard to check if a string is a valid price type
*/
export function isPriceType(pt) {
return validators['price:type'].test(pt);
}
/**
* Type guard to check if a string is a valid resolution
*/
export function isResolution(res) {
return validators['price:resolution'].test(res);
}
/**
* Type guard to check if a string is a valid changefreq value
*/
const CHANGEFREQ = Object.values(EnumChangefreq);
export function isValidChangeFreq(freq) {
return CHANGEFREQ.includes(freq);
}
/**
* Type guard to check if a string is a valid yes/no value
*/
export function isValidYesNo(yn) {
return /^(?:YES|NO|[Yy]es|[Nn]o)$/.test(yn);
}
/**
* Checks that a priority is a finite number within sitemaps.org bounds (0–1)
*/
export function isValidPriority(priority) {
return (typeof priority === 'number' &&
Number.isFinite(priority) &&
priority >= LIMITS.MIN_PRIORITY &&
priority <= LIMITS.MAX_PRIORITY);
}
/**
* Checks that a video duration is a finite number of seconds within Google's
* video sitemap bounds (0–28800)
*/
export function isValidVideoDuration(duration) {
return (isFinite(duration) &&
duration >= LIMITS.MIN_VIDEO_DURATION &&
duration <= LIMITS.MAX_VIDEO_DURATION);
}
/**
* Checks that a video rating is a finite number within Google's video sitemap
* bounds (0–5)
*/
export function isValidVideoRating(rating) {
return (isFinite(rating) &&
rating >= LIMITS.MIN_VIDEO_RATING &&
rating <= LIMITS.MAX_VIDEO_RATING);
}
/**
* Checks that a video view count is a finite, non-negative number
*/
export function isValidVideoViewCount(count) {
return isFinite(count) && count >= LIMITS.MIN_VIDEO_VIEW_COUNT;
}
/**
* Checks that a date string is in ISO 8601 / W3C format
*/
export function isValidISODate(date) {
return LIMITS.ISO_DATE_REGEX.test(date);
}
/**
* Returns an alternate link's language (`lang`, else `hreflang`), or
* `undefined` when neither is a non-empty string. Validation reports such a
* link, and the writer drops it.
*/
export function getLinkHreflang(link) {
const hreflang = link.lang || link.hreflang;
return typeof hreflang === 'string' && hreflang !== '' ? hreflang : undefined;
}
/**
* Type guard to check if a string is a valid allow/deny value
*/
export function isAllowDeny(ad) {
return allowDeny.test(ad);
}
/**
* Validates that a URL is well-formed and meets security requirements
*
* Security: This function enforces that URLs use safe protocols (http/https),
* are within reasonable length limits (2048 chars per sitemaps.org spec),
* and can be properly parsed. This prevents protocol injection attacks and
* ensures compliance with sitemap specifications.
*
* @param url - The URL to validate
* @param paramName - The parameter name for error messages
* @throws {InvalidURLError} If the URL is invalid (a subclass of InvalidHostnameError)
*/
export function validateURL(url, paramName) {
if (!url || typeof url !== 'string') {
throw new InvalidURLError(url, `${paramName} must be a non-empty string`);
}
if (url.length > LIMITS.MAX_URL_LENGTH) {
throw new InvalidURLError(url, `${paramName} exceeds maximum length of ${LIMITS.MAX_URL_LENGTH} characters`);
}
if (!LIMITS.URL_PROTOCOL_REGEX.test(url)) {
throw new InvalidURLError(url, `${paramName} must use http:// or https:// protocol`);
}
// Validate URL can be parsed
try {
new URL(url);
}
catch (err) {
throw new InvalidURLError(url, `${paramName} is not a valid URL: ${err instanceof Error ? err.message : String(err)}`);
}
}
/**
* Runs validateURL and returns its error message instead of throwing, for
* parsers that report invalid values and keep going.
*
* @returns The validation error message, or undefined if the URL is valid
*/
export function getURLError(url, paramName) {
try {
validateURL(url, paramName);
return undefined;
}
catch (error) {
return error instanceof Error ? error.message : String(error);
}
}
/**
* Validates that a path doesn't contain path traversal sequences
*
* Security: This function prevents path traversal attacks by detecting
* any occurrence of '..' in the path, whether it appears as '../', '/..',
* or standalone. This prevents attackers from accessing files outside
* the intended directory structure.
*
* @param path - The path to validate
* @param paramName - The parameter name for error messages
* @throws {InvalidPathError} If the path contains traversal sequences
*/
export function validatePath(path, paramName) {
if (!path || typeof path !== 'string') {
throw new InvalidPathError(path, `${paramName} must be a non-empty string`);
}
// Reject absolute paths to prevent arbitrary write location when caller input
// reaches destinationDir (BB-04)
if (isAbsolute(path)) {
throw new InvalidPathError(path, `${paramName} must be a relative path (absolute paths are not allowed)`);
}
// Check for path traversal sequences - must check before and after normalization
// to catch both Windows-style (\) and Unix-style (/) separators
if (path.includes('..')) {
throw new InvalidPathError(path, `${paramName} contains path traversal sequence (..)`);
}
// Additional check after normalization to catch encoded or obfuscated attempts
const normalizedPath = path.replace(/\\/g, '/');
const pathComponents = normalizedPath.split('/').filter((p) => p.length > 0);
if (pathComponents.includes('..')) {
throw new InvalidPathError(path, `${paramName} contains path traversal sequence (..)`);
}
// Check for null bytes (security issue in some contexts)
if (path.includes('\0')) {
throw new InvalidPathError(path, `${paramName} contains null byte character`);
}
}
/**
* Validates that a public base path is safe for URL construction
*
* Security: This function prevents path traversal attacks and validates
* that the path is safe for use in URL construction within sitemap indexes.
* It checks for '..' sequences, null bytes, and invalid whitespace that
* could be used to manipulate URL structure or inject malicious content.
*
* @param publicBasePath - The public base path to validate
* @throws {InvalidPublicBasePathError} If the path is invalid
*/
export function validatePublicBasePath(publicBasePath) {
if (!publicBasePath || typeof publicBasePath !== 'string') {
throw new InvalidPublicBasePathError(publicBasePath, 'must be a non-empty string');
}
// Check for path traversal - check the raw string first
if (publicBasePath.includes('..')) {
throw new InvalidPublicBasePathError(publicBasePath, 'contains path traversal sequence (..)');
}
// Additional check for path components after normalization
const normalizedPath = publicBasePath.replace(/\\/g, '/');
const pathComponents = normalizedPath.split('/').filter((p) => p.length > 0);
if (pathComponents.includes('..')) {
throw new InvalidPublicBasePathError(publicBasePath, 'contains path traversal sequence (..)');
}
// Check for null bytes
if (publicBasePath.includes('\0')) {
throw new InvalidPublicBasePathError(publicBasePath, 'contains null byte character');
}
// Check for potentially dangerous characters that could break URL construction
if (/[\r\n\t]/.test(publicBasePath)) {
throw new InvalidPublicBasePathError(publicBasePath, 'contains invalid whitespace characters');
}
}
/**
* Validates that a limit is within acceptable range per sitemaps.org spec
*
* Security: This function enforces sitemap size limits (1-50,000 URLs per
* sitemap) as specified by sitemaps.org. This prevents resource exhaustion
* attacks and ensures compliance with search engine requirements.
*
* @param limit - The limit to validate
* @throws {InvalidLimitError} If the limit is out of range
*/
export function validateLimit(limit) {
if (typeof limit !== 'number' ||
!Number.isFinite(limit) ||
Number.isNaN(limit)) {
throw new InvalidLimitError(limit);
}
if (limit < LIMITS.MIN_SITEMAP_ITEM_LIMIT ||
limit > LIMITS.MAX_SITEMAP_ITEM_LIMIT) {
throw new InvalidLimitError(limit);
}
// Ensure it's an integer
if (!Number.isInteger(limit)) {
throw new InvalidLimitError(limit);
}
}
/**
* Whether `n` is a valid parser entry cap: a non-negative integer, or
* `Infinity` for no cap. Unlike validateLimit, 0 and values above the
* sitemaps.org limit are allowed.
*/
export function isValidMaxEntries(n) {
return n === Infinity || (Number.isInteger(n) && n >= 0);
}
/**
* Validates that an XSL URL is safe and well-formed
*
* Security: This function validates XSL stylesheet URLs to prevent
* injection attacks. It blocks dangerous protocols and content patterns
* that could be used for XSS or other attacks. The validation uses
* case-insensitive matching to catch obfuscated attacks.
*
* @param xslUrl - The XSL URL to validate
* @throws {InvalidXSLUrlError} If the URL is invalid
*/
export function validateXSLUrl(xslUrl) {
if (!xslUrl || typeof xslUrl !== 'string') {
throw new InvalidXSLUrlError(xslUrl, 'must be a non-empty string');
}
if (xslUrl.length > LIMITS.MAX_URL_LENGTH) {
throw new InvalidXSLUrlError(xslUrl, `exceeds maximum length of ${LIMITS.MAX_URL_LENGTH} characters`);
}
if (!LIMITS.URL_PROTOCOL_REGEX.test(xslUrl)) {
throw new InvalidXSLUrlError(xslUrl, 'must use http:// or https:// protocol');
}
// Validate URL can be parsed
try {
new URL(xslUrl);
}
catch (err) {
throw new InvalidXSLUrlError(xslUrl, `is not a valid URL: ${err instanceof Error ? err.message : String(err)}`);
}
// Check for potentially dangerous content (case-insensitive)
const lowerUrl = xslUrl.toLowerCase();
// Block dangerous HTML/script content
if (lowerUrl.includes('<script')) {
throw new InvalidXSLUrlError(xslUrl, 'contains potentially malicious content (<script tag)');
}
// Block dangerous protocols (already checked http/https above, but double-check for encoded variants)
const dangerousProtocols = [
'javascript:',
'data:',
'vbscript:',
'file:',
'about:',
];
for (const protocol of dangerousProtocols) {
if (lowerUrl.includes(protocol)) {
throw new InvalidXSLUrlError(xslUrl, `contains dangerous protocol: ${protocol}`);
}
}
// Check for URL-encoded variants of dangerous patterns
// %3C = '<', %3E = '>', %3A = ':'
const encodedPatterns = [
'%3cscript', // <script
'%3c%73%63%72%69%70%74', // <script (fully encoded)
'javascript%3a', // javascript:
'data%3a', // data:
];
for (const pattern of encodedPatterns) {
if (lowerUrl.includes(pattern)) {
throw new InvalidXSLUrlError(xslUrl, 'contains URL-encoded malicious content');
}
}
// Reject unencoded XML special characters — these must be percent-encoded in
// valid URLs and could break out of XML attribute context if left raw.
if (xslUrl.includes('"') || xslUrl.includes('<') || xslUrl.includes('>')) {
throw new InvalidXSLUrlError(xslUrl, 'contains unencoded XML special characters (" < >); percent-encode them in the URL');
}
}
/**
* Internal helper to validate fields against their validators
*/
function validate(subject, url, level, errorHandler) {
Object.keys(subject).forEach((key) => {
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
const val = subject[key];
if (validators[key] && !validators[key].test(val)) {
errorHandler(new InvalidAttrValue(key, val, validators[key], url), level);
}
});
}
// Default for direct validateSMIOptions() calls: warn through the console
const handleError = loggingErrorHandler(defaultLogger);
/**
* Verifies all data passed in will comply with sitemap spec.
* @param conf Options to validate
* @param level logging level
* @param errorHandler error handling func
*/
export function validateSMIOptions(conf, level = ErrorLevel.WARN, errorHandler = handleError) {
if (!conf) {
throw new NoConfigError();
}
if (level === ErrorLevel.SILENT) {
return conf;
}
const { url, changefreq, priority, news, video } = conf;
if (!url) {
errorHandler(new NoURLError(), level);
}
if (changefreq) {
if (!isValidChangeFreq(changefreq)) {
errorHandler(new ChangeFreqInvalidError(url, changefreq), level);
}
}
if (priority !== undefined && priority !== null) {
if (!isValidPriority(priority)) {
errorHandler(new PriorityInvalidError(url, priority), level);
}
}
if (Array.isArray(conf.links)) {
conf.links.forEach((link) => {
if (getLinkHreflang(link) === undefined) {
errorHandler(new InvalidLinkHreflang(url, link.url, link.lang || link.hreflang), level);
}
});
}
if (news) {
if (news.access &&
news.access !== 'Registration' &&
news.access !== 'Subscription') {
errorHandler(new InvalidNewsAccessValue(url, news.access), level);
}
if (!news.publication ||
!news.publication.name ||
!news.publication.language ||
!news.publication_date ||
!news.title) {
errorHandler(new InvalidNewsFormat(url), level);
}
validate(news, url, level, errorHandler);
validate(news.publication, url, level, errorHandler);
}
if (video) {
video.forEach((vid) => {
if (vid.duration !== undefined) {
if (!isValidVideoDuration(vid.duration)) {
errorHandler(new InvalidVideoDuration(url, vid.duration), level);
}
}
if (vid.rating !== undefined && !isValidVideoRating(vid.rating)) {
errorHandler(new InvalidVideoRating(url, vid.title, vid.rating), level);
}
if (typeof vid !== 'object' ||
!vid.thumbnail_loc ||
!vid.title ||
!vid.description) {
// has to be an object and include required categories https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190
errorHandler(new InvalidVideoFormat(url), level);
}
if (vid.title.length > 100) {
errorHandler(new InvalidVideoTitle(url, vid.title.length), level);
}
if (vid.description.length > 2048) {
errorHandler(new InvalidVideoDescription(url, vid.description.length), level);
}
if (vid.view_count !== undefined &&
!isValidVideoViewCount(vid.view_count)) {
errorHandler(new InvalidVideoViewCount(url, vid.view_count), level);
}
if (vid.tag.length > 32) {
errorHandler(new InvalidVideoTagCount(url, vid.tag.length), level);
}
if (vid.category !== undefined && vid.category?.length > 256) {
errorHandler(new InvalidVideoCategory(url, vid.category.length), level);
}
if (vid.family_friendly !== undefined &&
!isValidYesNo(vid.family_friendly)) {
errorHandler(new InvalidVideoFamilyFriendly(url, vid.family_friendly), level);
}
for (const field of [
'requires_subscription',
'live',
'player_loc:allow_embed',
]) {
const value = vid[field];
if (value !== undefined && !isValidYesNo(value)) {
errorHandler(new InvalidVideoYesNo(url, field, value), level);
}
}
if (vid.restriction) {
if (!validators.restriction.test(vid.restriction)) {
errorHandler(new InvalidVideoRestriction(url, vid.restriction), level);
}
if (!vid['restriction:relationship'] ||
!isAllowDeny(vid['restriction:relationship'])) {
errorHandler(new InvalidVideoRestrictionRelationship(url, vid['restriction:relationship']), level);
}
}
// TODO price element should be unbounded
if ((vid.price === '' && vid['price:type'] === undefined) ||
(vid['price:type'] !== undefined && !isPriceType(vid['price:type']))) {
errorHandler(new InvalidVideoPriceType(url, vid['price:type'], vid.price), level);
}
if (vid['price:resolution'] !== undefined &&
!isResolution(vid['price:resolution'])) {
errorHandler(new InvalidVideoResolution(url, vid['price:resolution']), level);
}
if (vid['price:currency'] !== undefined &&
!validators['price:currency'].test(vid['price:currency'])) {
errorHandler(new InvalidVideoPriceCurrency(url, vid['price:currency']), level);
}
validate(vid, url, level, errorHandler);
});
}
return conf;
}
+14
View File
@@ -0,0 +1,14 @@
import { Readable } from 'node:stream';
/**
* Verify the passed in xml is valid. Requires xmllib be installed
*
* Security: This function always pipes XML content via stdin to prevent
* command injection vulnerabilities. Never pass user-controlled strings
* as file path arguments to xmllint.
*
* @param xml what you want validated (string or Readable stream)
* @return {Promise<void>} resolves on valid. Rejects with [error, stderr] when
* the xml is invalid, or with [error] alone when validation could not run
* (xmllint missing or not executable, or the schema not found).
*/
export declare function xmlLint(xml: string | Readable): Promise<void>;
+106
View File
@@ -0,0 +1,106 @@
import { Readable } from 'node:stream';
import { execFile } from 'node:child_process';
import { XMLLintUnavailable } from './errors.js';
import { findUp } from './utils.js';
/**
* Finds `schema/all.xsd`, which ships with this package.
*
* Walks up from this module's own directory rather than `process.cwd()`, so
* the lookup works however the code is run: from source (`lib/xmllint.ts`),
* from the build (`dist/lib/xmllint.js`), or installed as a dependency
* (`node_modules/sitemap/dist/lib/xmllint.js`), whatever the caller's working
* directory is. `cli.ts` finds `package.json` the same way.
*
* @throws {Error} if the schema is not found
* @returns {string} the path to `all.xsd`
*/
function findSchema() {
const schema = findUp('schema/all.xsd', import.meta.dirname);
if (!schema) {
throw new Error(`Schema not found: no schema/all.xsd in ${import.meta.dirname} or any parent directory`);
}
return schema;
}
/**
* Verify the passed in xml is valid. Requires xmllib be installed
*
* Security: This function always pipes XML content via stdin to prevent
* command injection vulnerabilities. Never pass user-controlled strings
* as file path arguments to xmllint.
*
* @param xml what you want validated (string or Readable stream)
* @return {Promise<void>} resolves on valid. Rejects with [error, stderr] when
* the xml is invalid, or with [error] alone when validation could not run
* (xmllint missing or not executable, or the schema not found).
*/
export function xmlLint(xml) {
return new Promise((resolve, reject) => {
// Inside the executor so a missing schema rejects (matching the
// [error] shape callers expect) instead of throwing synchronously
let schema;
try {
schema = findSchema();
}
catch (err) {
reject([err]);
return;
}
const args = [
'--schema',
schema,
'--noout',
'-', // Always read from stdin for security
];
// Spawn xmllint directly rather than probing with `which` first: `which`
// is not a command on Windows, so probing there reported xmllint as
// unavailable even when it was installed. A missing binary surfaces as
// ENOENT from the spawn itself on every platform, and this saves a process.
const xmllint = execFile('xmllint', args, (error, stdout, stderr) => {
// If xmllint exited (or never started) before reading all of a stream
// input, stop the source so it doesn't sit paused holding a file
// descriptor open after the promise settles.
if (typeof xml !== 'string' && xml && !xml.readableEnded) {
if (xmllint.stdin) {
xml.unpipe(xmllint.stdin);
}
xml.destroy();
}
if (error) {
const { code, syscall } = error;
if (code === 'ENOENT') {
reject([new XMLLintUnavailable()]);
}
else if (syscall?.startsWith('spawn')) {
// xmllint could not be started at all (such as EACCES), so there
// is no validation output: reject without stderr, like a missing
// schema, so callers don't report the file as invalid
reject([error]);
}
else {
reject([error, stderr]);
}
return;
}
resolve();
});
// Always pipe XML content via stdin for security
if (xmllint.stdin) {
// When the binary is missing the stdin socket is torn down immediately,
// so writing to it emits EPIPE. The execFile callback above reports the
// real failure; don't let that stray error go unhandled and crash.
xmllint.stdin.on('error', () => undefined);
if (typeof xml === 'string') {
// Convert string to stream and pipe to stdin
xmllint.stdin.write(xml);
xmllint.stdin.end();
}
else if (xml) {
// Pipe readable stream to stdin
xml.pipe(xmllint.stdin);
}
}
if (xmllint.stdout) {
xmllint.stdout.unpipe();
}
});
}
+93
View File
@@ -0,0 +1,93 @@
{
"name": "sitemap",
"version": "10.0.0",
"description": "Sitemap-generating lib/cli",
"keywords": [
"sitemap",
"sitemap.xml"
],
"homepage": "https://github.com/ekalinin/sitemap.js#readme",
"bugs": {
"url": "https://github.com/ekalinin/sitemap.js/issues"
},
"repository": {
"type": "git",
"url": "git+https://github.com/ekalinin/sitemap.js.git"
},
"license": "MIT",
"author": "Eugene Kalinin <e.v.kalinin@gmail.com>",
"type": "module",
"exports": {
".": {
"types": "./dist/index.d.ts",
"default": "./dist/index.js"
},
"./package.json": "./package.json"
},
"module": "./dist/index.js",
"types": "./dist/index.d.ts",
"bin": "./dist/cli.js",
"directories": {
"lib": "lib",
"test": "tests"
},
"files": [
"dist",
"schema"
],
"scripts": {
"build": "tsc -p tsconfig.build.json",
"lint": "eslint \"{lib,tests}/**/*.ts\" ./cli.ts",
"lint:fix": "eslint --fix \"{lib,tests}/**/*.ts\" ./cli.ts",
"prepare": "husky",
"prepublishOnly": "rm -rf dist && npm run build && npm run test",
"prettier": "npx prettier --check \"{lib,tests}/**/*.ts\" ./cli.ts",
"prettier:fix": "npx prettier --write \"{lib,tests}/**/*.ts\" ./cli.ts",
"test": "npm run build && node --experimental-strip-types --test tests/*.test.ts",
"test:coverage": "npm run build && node --experimental-strip-types --experimental-test-coverage --test-coverage-branches=80 --test-coverage-functions=90 --test-coverage-lines=90 --test tests/*.test.ts",
"test:full": "npm run lint && npm run test:coverage && npm run test:xmllint",
"test:perf": "node ./tests/perf.mjs",
"test:schema": "node ./tests/xmllint-schema.mjs",
"test:typecheck": "tsc --noEmit",
"test:xmllint": "node ./tests/xmllint-schema.mjs --skip-if-missing"
},
"lint-staged": {
"package.json": [
"sort-package-json"
],
"{lib,tests}/**/*.ts": [
"eslint --fix",
"prettier --write"
]
},
"dependencies": {
"@types/node": "^22.0.0 || ^24.0.0 || ^26.0.0",
"@types/sax": "^1.2.1",
"sax": "^1.4.1"
},
"devDependencies": {
"@eslint/eslintrc": "^3.3.1",
"@eslint/js": "^10.0.1",
"@types/memorystream": "^0.3.0",
"@typescript-eslint/eslint-plugin": "^8.46.2",
"@typescript-eslint/parser": "^8.46.2",
"eslint": "^10.13.0",
"eslint-config-prettier": "^10.1.8",
"eslint-plugin-prettier": "^5.1.3",
"globals": "^17.13.0",
"husky": "^9.0.11",
"lint-staged": "^17.6.0",
"memorystream": "^0.3.1",
"prettier": "^3.2.5",
"sort-package-json": "^4.1.0",
"stats-lite": "^2.2.0",
"stream-json": "^3.7.0",
"through2-map": "^4.0.0",
"typescript": "^5.4.5"
},
"packageManager": "npm@11.17.0",
"engines": {
"node": ">=22.22.1",
"npm": ">=10.8.2"
}
}
+8
View File
@@ -0,0 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<schema elementFormDefault="qualified" xmlns="http://www.w3.org/2001/XMLSchema">
<import namespace="http://www.sitemaps.org/schemas/sitemap/0.9" schemaLocation="./sitemap.xsd" />
<import namespace="http://www.google.com/schemas/sitemap-video/1.1" schemaLocation="http://www.google.com/schemas/sitemap-video/1.1/sitemap-video.xsd"/>
<import namespace="http://www.google.com/schemas/sitemap-image/1.1" schemaLocation="http://www.google.com/schemas/sitemap-image/1.1/sitemap-image.xsd"/>
<import namespace="http://www.google.com/schemas/sitemap-news/0.9" schemaLocation="http://www.google.com/schemas/sitemap-news/0.9/sitemap-news.xsd"/>
<import namespace="http://www.w3.org/1999/xhtml" schemaLocation="http://www.w3.org/2002/08/xhtml/xhtml1-strict.xsd"/>
</schema>
+116
View File
@@ -0,0 +1,116 @@
<?xml version="1.0" encoding="UTF-8"?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
targetNamespace="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
elementFormDefault="qualified">
<xsd:annotation>
<xsd:documentation>
XML Schema for Sitemap files.
Last Modifed 2008-03-26
</xsd:documentation>
</xsd:annotation>
<xsd:element name="urlset">
<xsd:annotation>
<xsd:documentation>
Container for a set of up to 50,000 document elements.
This is the root element of the XML file.
</xsd:documentation>
</xsd:annotation>
<xsd:complexType>
<xsd:sequence>
<xsd:any namespace="##other" minOccurs="0" maxOccurs="unbounded" processContents="strict"/>
<xsd:element name="url" type="tUrl" maxOccurs="unbounded"/>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
<xsd:complexType name="tUrl">
<xsd:annotation>
<xsd:documentation>
Container for the data needed to describe a document to crawl.
</xsd:documentation>
</xsd:annotation>
<xsd:sequence>
<xsd:element name="loc" type="tLoc"/>
<xsd:element name="lastmod" type="tLastmod" minOccurs="0"/>
<xsd:element name="changefreq" type="tChangeFreq" minOccurs="0"/>
<xsd:element name="priority" type="tPriority" minOccurs="0"/>
<xsd:any namespace="##other" minOccurs="0" maxOccurs="unbounded" processContents="strict"/>
</xsd:sequence>
</xsd:complexType>
<xsd:simpleType name="tLoc">
<xsd:annotation>
<xsd:documentation>
REQUIRED: The location URI of a document.
The URI must conform to RFC 2396 (http://www.ietf.org/rfc/rfc2396.txt).
</xsd:documentation>
</xsd:annotation>
<xsd:restriction base="xsd:anyURI">
<xsd:minLength value="12"/>
<xsd:maxLength value="2048"/>
</xsd:restriction>
</xsd:simpleType>
<xsd:simpleType name="tLastmod">
<xsd:annotation>
<xsd:documentation>
OPTIONAL: The date the document was last modified. The date must conform
to the W3C DATETIME format (http://www.w3.org/TR/NOTE-datetime).
Example: 2005-05-10
Lastmod may also contain a timestamp.
Example: 2005-05-10T17:33:30+08:00
</xsd:documentation>
</xsd:annotation>
<xsd:union>
<xsd:simpleType>
<xsd:restriction base="xsd:date"/>
</xsd:simpleType>
<xsd:simpleType>
<xsd:restriction base="xsd:dateTime"/>
</xsd:simpleType>
</xsd:union>
</xsd:simpleType>
<xsd:simpleType name="tChangeFreq">
<xsd:annotation>
<xsd:documentation>
OPTIONAL: Indicates how frequently the content at a particular URL is
likely to change. The value "always" should be used to describe
documents that change each time they are accessed. The value "never"
should be used to describe archived URLs. Please note that web
crawlers may not necessarily crawl pages marked "always" more often.
Consider this element as a friendly suggestion and not a command.
</xsd:documentation>
</xsd:annotation>
<xsd:restriction base="xsd:string">
<xsd:enumeration value="always"/>
<xsd:enumeration value="hourly"/>
<xsd:enumeration value="daily"/>
<xsd:enumeration value="weekly"/>
<xsd:enumeration value="monthly"/>
<xsd:enumeration value="yearly"/>
<xsd:enumeration value="never"/>
</xsd:restriction>
</xsd:simpleType>
<xsd:simpleType name="tPriority">
<xsd:annotation>
<xsd:documentation>
OPTIONAL: The priority of a particular URL relative to other pages
on the same site. The value for this element is a number between
0.0 and 1.0 where 0.0 identifies the lowest priority page(s).
The default priority of a page is 0.5. Priority is used to select
between pages on your site. Setting a priority of 1.0 for all URLs
will not help you, as the relative priority of pages on your site
is what will be considered.
</xsd:documentation>
</xsd:annotation>
<xsd:restriction base="xsd:decimal">
<xsd:minInclusive value="0.0"/>
<xsd:maxInclusive value="1.0"/>
</xsd:restriction>
</xsd:simpleType>
</xsd:schema>