Here's a basic template example, which utilizes Puppeteer's Chromium browser in headless mode. It processes a list of websites in separate threads, by opening each one in Chromium and saving its content in a text file. What the template does by step:
- Provides user with 2 settings: "create a file" dialog button and text input with pre-filled websites list in config.userSettings.
- Generates tasks in generateTasks function, which contain a website address and a target file name.
- OpenSubmitter runs each task in a separate thread, where it launches the template code and feeds task into runTask function.
- The runTask function uses Puppeteer's page object to navigate to the website, retrieve its content and save it to the text file, included in the task at the 2nd step.
/// <reference path="type.d.ts" />
const fs = require("fs")
class Template implements OpenSubmitterTemplateProtocol {
config: TemplateConfig = {
// Name and description to display in OpenSubmitter's UI:
name: 'Basic Template Example (puppeteer)',
description: 'Open a websites with puppeteer-chromium in headless mode and write output to different files',
// Based on this setting, OpenSubmitter will inject Puppeteer's page object into this template
capabilities: ['puppeteer'],
// This tells OpenSubmitter that the user is allowed to specify amount of threads:
multiThreadingEnabled: true,
// User's settings for OpenSubmitter UI:
userSettings: [
{
// A text input with a button which opens "create a file" dialog
type: 'OutputFile',
name: 'outputFile',
title: 'Where to write the output of the download',
fileName: "",
required: false,
uiWidth: 100
},{
// Multi-lined text input with pre-filled value
type: 'Textarea',
name: 'websites',
title: 'Websites list, one per line',
value: 'https://www.google.com/\nhttps://www.github.com/\nhttps://www.bing.com/'
}
]
};
// Dummy variable, will be overridden by OpenSubmitter with Puppeteer's page object
page = null;
// This method should generate tasks before OpenSubmitter starts executing them
async generateTasks(...args: any): Promise<TemplateTask[]> {
// "OutputFile" user settings value from the config above
const fileName = this.config.userSettings.find(setting => setting.name === 'outputFile').fileName;
// "Textarea" user settings value from the config
const websiteValue = this.config.userSettings.find(setting => setting.name === 'websites').value;
// Splitting websites list by new line
const websitesList = websiteValue.toString().split("\n")
const result: TemplateTask[] = [];
// Iterating through websites list
for (const website of websitesList) {
// Validate the website address
if (!/^(https?:\/\/)?([\da-z.-]+)\.([a-z.]{2,6})([\/\w .-]*)*\/?$/.test(website)) continue;
// Add website's address and unique text file name to the task
result.push({
data: {
url: website,
// If file path is specified, return file path + website address without special characters
// Otherwise have an empty fileName
fileName: fileName && fileName.length > 0 ?
(fileName + "." + website.replace(/[^a-zA-Z0-9]/g, '') + ".txt") :
null
}
})
}
return result;
}
// Running the task previously generated in "generateTasks" function
// Each task is executed in a separate thread.
async runTask(task: TemplateTask) {
// Navigate to the task's page
try {
this.log(`navigating to ${task.data.url}...`);
await this.page.goto(task.data.url, {
waitUntil: "networkidle0",
timeout: 20000
});
} catch (e) {
this.log('err while loading the page: ' + e);
}
// Get the HTML contents of the page
const result = await this.page.content();
// If user has provided an output file path, save contents there
if (task.data.fileName) {
fs.writeFileSync(task.data.fileName, result);
}
}
log(msg: string) {
console.log(msg);
}
}
How the whole process works
1. Your templates exposes configuration in config property. Mostly it programs how an end-user is interacting with your template: they may change some settings, point to files, etc.
2. When the user clicks on "Run template", OpenSubmitter compiles it from Typescript to Javascript and runs generateTasks method. This method should generate an array of tasks which OpenSubmitter keeps in memory.
3. OpenSubmitter starts a thread pool, in each thread it creates a new instance of your template class and feeds a task via runTask method. When the method exits the thread is re-used for the next task, until all tasks are processed.
4. If you'd disabled multi-threading in your template with config.multiThreadingEnabled = false, then all the tasks are processed in a single thread.
UI settings example
This is how the above template's config.userSettings look in UI. Notice how the config.name, config.description, config.multiThreadingEnabled and config.userSettings are rendered in the UI.
Where to start
The best way to start programming your own templates for OpenSubmitter is the following:
1. Clone the repository.
2. Run npm install && npm run dev .
3. Use any example in templates directory as starting point to create your own code. Make a copy of any template, change config.name property and you're ready to create!