Web scraper IDE - Coding Environment and Tutorials

  • IDE Best Practices

    Best practices Missing dead_page condition. When using navigate command, a 'dead_page' condition should be added to check if the page is not found. This will prevent automatic retries. While we automatically handle this when the r...

  • IDE Complete examples

    Complete examples Introduction The collect and parse commands have been removed. The data will be returned from parser code as an object or array, and it will be automatically saved to the output: // Interaction code // Old code n...

  • IDE Page

    A : See more examples - Examples of template code that our collector engineers built. B : Add another step (stage) - It is useful to add stages when you want to collect data from multiple pages. For example, in case you want to co...

  • Coding environment - IDE Interaction code

    These are all of the codes that you can do with the IDE input - Global object available to the interaction code. Provided by trigger input or next_stage() calls navigate(input.url); navigate - Navigate the browser session to a UR...

  • Coding environment - IDE Parser code

    These are all of the codes that you can do with the IDE input - Global variable available to the parser code let url = input.url; $ - An instance of cheerioFind more information on the cheerio website. $('#example').text() $('$ex...

  • Finding element selectors

    In order to target an element (to click it or pull text out), you need to specify the element with a CSS selector. A CSS selector can match one or more items on the page. Most commands in the interaction code will require a selecto...

  • Building element selectors

    Selectors are built out of 4 basic components: p : the element type selector. This example will match any <p></p> element on the page [href] : square brackets is an attribute selector. This example will match any element with the ...

  • jQuery expressions

    You can use jQuery-like expressions in interaction code. For instance: wait($('selector')); // wait for this element to appear click($('selector')); // click on this element after it appears // wait for an element that matches the ...