Code Snippets

A simple bash-script to crawl your site and follow all links and generate a sitemap. | bash

You can create multiple sitemaps with this. Note, I don't set a priority of anything fancy.

NOTE: I heavily modified a script from somewhere else. But I don't recall where from. If you are the author feel free to get in contact for a link.


function kit_sitemap() {
	local sitedomain="${1}"
	local sitemapfile="${2}"

	#echo 'Doing sitemap - '$sitedomain' > '$sitemapfile

	wget --reject-regex "(.*)\?(.*)" --spider --no-verbose --recursive --level=inf --output-file=/var/app/current/sitemaps/linklist.txt $sitedomain
	grep -i URL /var/app/current/sitemaps/linklist.txt | awk -F 'URL:' '{print $2}' | awk '{$1=$1};1' | awk '{print $1}' | sort -u | sed '/^$/d' > /var/app/current/sitemaps/sortedurls.txt
	header='<?xml version="1.0" encoding="UTF-8"?><urlset
		  xmlns="http:// //"
		  xmlns:xsi="http:// //"
		  xsi:schemaLocation="http:// //
				http:// //">'
	echo $header > $sitemapfile
	while read p; do
	  case "$p" in
	  */ | *.html | *.htm | *.php)
		echo '<url><loc>'$p'</loc></url>' >> $sitemapfile
	done < /var/app/current/sitemaps/sortedurls.txt
	echo "</urlset>" >> $sitemapfile
	echo 'Done: '$sitedomain

sitedomain=https:// //
kit_sitemap $sitedomain $sitemapfile

sitedomain=https:// //
kit_sitemap $sitedomain $sitemapfile

# etc....

# email admin
echo "completed from _cron/daily/" | mail -s " done" ""

Published: Sun 24th December 2017
||COMMENTS|| This site proudly uses PrismJS to display code snippets

If you like this page, please share it.

Code Links - PHP, JS, CSS, Bootstrap, Bash, Emmet, IntelliJ, Sublime etc.

My personal list of links which I keep here publicly incase they help someone else one day.

Code Snippets

Snippets of PHP code I have gathered over the years. I keep them here for my own repository but also to share with others. I always link back when I remember the source.

My Github Repos

A work in progress. I am going to make as many modules that I write in my devbox public as possible.

Kit's Homepage

This site was originally intended as a test bed for code. It now includes my blog about business and powerlifting, as well as games for Spanish language learning

© 2018