SiteCrawler
Used in:
components
- OverviewOverview
- VersionsVersions
- DependentsDependents
- DependenciesDependencies
<dependency>
<groupId>io.github.jasperroel</groupId>
<artifactId>SiteCrawler</artifactId>
<version>1.0.0</version>
</dependency><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>io.github.jasperroel</groupId>
<artifactId>SiteCrawler</artifactId>
<version>1.0.0</version>
<packaging>jar</packaging>
<name>SiteCrawler</name>
<description>This project provides a simple WebCrawler with retry-capabilities, functionality to distinguish between http/https sites.
It biggest feature is that it allows for plugins (or CrawlerActions), which allows you to hook your scripts into the crawling process.
It also allow for setting "blocked" URLs. Those URLs or patterns will not be crawled.</description>
<inceptionYear>Feb 1, 2014</inceptionYear>
<organization>
<name>Salesforce.com</name>
<url>https://www.salesforce.com</url>
</organization>
<developers>
<developer>
<name>Jasper Roel</name>
<email>jasper@jasperroel.nl</email>
</developer>
</developers>
<scm>
<url>https://github.com/forcedotcom/SiteCrawler</url>
<developerConnection>Jasper Roel</developerConnection>
</scm>
<url>https://github.com/forcedotcom/SiteCrawler</url>
<licenses>
<license>
<name>The BSD 2-Clause License</name>
<url>https://opensource.org/licenses/BSD-2-Clause</url>
</license>
</licenses>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<distributionManagement>
<snapshotRepository>
<id>ossrh</id>
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
</snapshotRepository>
<repository>
<id>ossrh</id>
<url>https://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
</repository>
</distributionManagement>
<dependencies>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
<version>1.7.10</version>
</dependency>
<dependency>
<groupId>net.sourceforge.htmlunit</groupId>
<artifactId>htmlunit</artifactId>
<version>2.17</version>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.6</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.1</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
<!--<plugin>-->
<!--<!– NOTE: We don't need a groupId specification because the group is-->
<!--org.apache.maven.plugins ...which is assumed by default.-->
<!--–>-->
<!--<artifactId>maven-assembly-plugin</artifactId>-->
<!--<version>2.6</version>-->
<!--<configuration>-->
<!--<descriptorRefs>-->
<!--<descriptorRef>jar-with-dependencies</descriptorRef>-->
<!--</descriptorRefs>-->
<!--</configuration>-->
<!--<executions>-->
<!--<execution>-->
<!--<id>make-assembly</id> <!– this is used for inheritance merges –>-->
<!--<phase>package</phase> <!– bind to the packaging phase –>-->
<!--<goals>-->
<!--<goal>single</goal>-->
<!--</goals>-->
<!--</execution>-->
<!--</executions>-->
<!--</plugin>-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.8.2</version>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<groupId>org.sonatype.plugins</groupId>
<artifactId>nexus-staging-maven-plugin</artifactId>
<version>1.6.8</version>
<extensions>true</extensions>
<configuration>
<!-- we only skip staging for non-ossrh repositories
<skipStaging>true</skipStaging>
-->
<serverId>ossrh</serverId>
<nexusUrl>https://oss.sonatype.org/</nexusUrl>
<!-- set this to true for auto release, on false you can inspect
and do a manual mvn nexus-staging:release -->
<autoReleaseAfterClose>false</autoReleaseAfterClose>
</configuration>
<executions>
<execution>
<id>default-deploy</id>
<phase>deploy</phase>
<goals>
<goal>deploy</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-gpg-plugin</artifactId>
<version>1.6</version>
<executions>
<execution>
<id>sign-artifacts</id>
<phase>verify</phase>
<goals>
<goal>sign</goal>
</goals>
<configuration>
<keyname>${gpg.keyname}</keyname>
<passphraseServerId>${gpg.keyname}</passphraseServerId>
<gpgArguments>
<arg>--pinentry-mode</arg>
<arg>loopback</arg>
</gpgArguments>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>2.2.1</version>
<executions>
<execution>
<id>attach-sources</id>
<goals>
<goal>jar-no-fork</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>3.0.1</version>
<executions>
<execution>
<id>attach-javadocs</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<reporting>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>versions-maven-plugin</artifactId>
<version>2.5</version>
<reportSets>
<reportSet>
<reports>
<report>dependency-updates-report</report>
<report>plugin-updates-report</report>
<report>property-updates-report</report>
</reports>
</reportSet>
</reportSets>
</plugin>
</plugins>
</reporting>
</project>
Open source consumption has exploded, with estimates placing this year’s downloads at over 6.6 trillion. This reliance on open source components, now making up to 90% of the modern software application, has ushered in both unprecedented innovation and complex challenges for software supply chains.
View the Report