Help files

Web Data Extraction Part III

Very often we want to extract data from a web page after a search, which has returned more than one page of results. In this case we want to perform the extraction to all, or to a number of pages.

This can be achieved with the "Extract Data from Web Page" action, if you select to set an element as "Pager". As shown below, you can right click on the pager element and select "Set This Element As Pager".

Having done so, upon extraction of the data, the Process will keep clicking on the "next page" button and extracting what you have set it to extract, up until when the pager element is enabled and present on the web page.


The CSS selector for the pager is shown in the "Advanced Settings" for the Extraction Preview window. You can modify it and use your own if you are familiar with CSS selector and maybe use its id and class attributes to make it 100% consistent.

Once you have set the data to extract, press "Commit". Back in the "Extract Data from Web Page" action you can choose to extract the results from:

- All Available pages, which means that the extraction will finish once there are no more result pages, or

- only first 'X' Web Pages, which means that the extraction will finish after extracting the data from the X web page.


Feel free to have a look at the Create CSS Selectors topic.